Table of Contents

UNITED STATES

SECURITIES AND EXCHANGE COMMISSION

WASHINGTON, D.C. 20549

FORM 10-K

SECURITIES EXCHANGE ACT OF 1934

For the fiscal year ended January 31, 2004 (Fiscal 2003)

Commission File Number 0-15898

CASUAL MALE RETAIL GROUP, INC.

(Exact name of registrant as specified in its charter)

(781) 828-9300

(Registrant’s telephone number, including area code)

Securities registered pursuant to Section 12(b) of the Act:

None

Securities registered pursuant to Section 12(g) of the Act:

Common Stock, $0.01 par value

(Title of each Class)

Indicate by check mark whether the registrant (1) has filed all reports required to be filed by Section 13 or 15(d) of the Securities Exchange Act of 1934 during the preceding 12 months (or for such shorter period that the registrant was required to file such reports), and (2) has been subject to such filing requirements for the past 90 days.    Yes  x    No  ¨

Indicate by check mark if disclosure of delinquent filers pursuant to Item 405 of Regulation S-K is not contained herein, and will not be contained, to the best of the registrant’s knowledge, in definitive proxy or information statements incorporated by reference in Part III of this Form 10-K or any amendment to this Form 10-K.    ¨

Indicate by check mark whether the registrant is an accelerated filer (as defined in Rule 12b-2 of the Act).    Yes  x    No  ¨

As of August 2, 2003, the aggregate market value of the Common Stock held by non-affiliates of the registrant was approximately $184.9 million, based on the last reported sale price on that date. Shares of Common Stock held by each executive officer and director and by each person who owns 10% or more of the outstanding Common Stock have been excluded on the basis that such persons may be deemed to be affiliates. This determination of affiliate status is not necessarily determinative for other purposes.

The registrant had 35,099,382 shares of Common Stock, $0.01 par value, outstanding as of March 31, 2004.

DOCUMENTS INCORPORATED BY REFERENCE

None.

Table of Contents

CASUAL MALE RETAIL GROUP, INC.

Index to Annual Report on Form 10-K

Year Ended January 31, 2004

2

Table of Contents

PART I.

Certain statements contained in this Annual Report on Form 10-K constitute “forward-looking statements” within the meaning of the United States Private Securities Litigation Reform Act of 1995. In some cases, forward-looking statements can be identified by the use of forward-looking terminology such as “may,” “will,” “estimate,” “intend,” “plan,” “continue,” “believe,” “expect” or “anticipate” or the negatives thereof, variations thereon or similar terminology. The forward-looking statements contained in this Annual Report are generally located in the material set forth under the headings “Business” and “Management’s Discussion and Analysis of Financial Condition and Results of Operations,” but may be found in other locations as well. These forward-looking statements generally relate to plans and objectives for future operations and are based upon management’s reasonable estimates of future results or trends. The forward-looking statements in this Annual Report should not be regarded as a representation by the Company or any other person that the objectives or plans of the Company will be achieved. Numerous factors could cause the Company’s actual results to differ materially from such forward-looking statements. The Company encourages readers to refer to the Company’s Current Report on Form 8-K, previously filed with the Securities and Exchange Commission on April 14, 2004, which identifies certain risks and uncertainties that may have an impact on future earnings and the direction of the Company.

All subsequent written and oral forward-looking statements attributable to the Company or to persons acting on the Company’s behalf are expressly qualified in their entirety by the foregoing. These forward-looking statements speak only as of the date of the document in which they are made. The Company disclaims any obligation or undertaking to provide any updates or revisions to any forward-looking statement to reflect any change in its expectations or any change in events, conditions or circumstances in which the forward-looking statement is based.

Casual Male Retail Group, Inc. together with its subsidiaries (the “Company”) is the largest specialty retailer of big and tall men’s apparel in the United States. The Company operates 481 Casual Male Big & Tall stores, the Casual Male catalog business and two e-commerce websites, 58 Levi’s^®/Dockers^® Outlet by Designs and 21 Ecko Unltd.^® outlet stores, all of which are located throughout the United States and Puerto Rico. Unless the context indicates otherwise, all references to “we,” “our,” “ours,” “us” and “the Company” refer to Casual Male Retail Group, Inc. and its consolidated subsidiaries. The Company refers to its fiscal years ended January 31, 2004, February 1, 2003 and February 2, 2002 as “fiscal 2003”, “fiscal 2002” and “fiscal 2001”, respectively.

HISTORY

The Company was incorporated in the State of Delaware in 1976 under the name Designs, Inc. Until fiscal 1995, the Company operated exclusively Levi Strauss & Co. branded apparel mall and outlet stores. In fiscal 1995, the Company began seeing limited growth opportunities with Levi Strauss & Co. and started to embark on several private label diversification strategies. These strategies ultimately were abandoned by the Company due to a variety of reasons, including lack of brand recognition of its private label brands by its customers. As a result of these failed strategies, the Company incurred approximately $85.6 million in operating losses during fiscal years 1997, 1998 and 1999.

In October 1999, the stockholders of the Company elected a new Board of Directors, which subsequently appointed a new Chairman of the Board of Directors of the Company and management team. Under this new management, the Company saw significant cost reductions in both its store and overhead operations, resulting in a return to profitability in fiscal 2000. With this new effective low-cost structure in place, the Company renewed its strategy to become the premier operator in the outlet channel for other well-known branded manufacturers, including Candie’s Inc., a leading designer and marketer of young women’s footwear, apparel and accessories. In April 2002, the Company entered into a joint venture arrangement with Ecko.Complex, LLC (“Ecko”), a leading design-driven lifestyle brand targeting young men and women with worldwide annual sales exceeding $200 million. Under this joint venture arrangement, the Company plans to exclusively open and operate 75 Ecko Unltd.^® branded outlet stores throughout the United States over a six-year period.

While implementing these initiatives, but with limited opportunity to expand its mature Levi’s^®/Dockers^® business, in May 2002 the Company acquired the Casual Male business from Casual Male Corp. at a bankruptcy court-ordered auction.

3

Table of Contents

At the time of the acquisition, Casual Male was the largest retailer of men’s clothing in the big and tall market in the United States. The Company acquired substantially all of the assets of Casual Male and certain of its subsidiaries for a purchase price of approximately $170 million, plus assumption of certain operating liabilities. In view of the significance of this acquisition to the growth and future identity of the Company, on August 8, 2002, the Company’s stockholders voted to change the Company’s name to “Casual Male Retail Group, Inc.”

Following the Casual Male acquisition, the Company re-evaluated its strategic initiatives. In light of the significant opportunity to grow the Casual Male business and the continued significant deterioration in its Levi’s^®/Dockers^® business, in fiscal 2002 the Company announced that it would downsize and eventually exit the Levi’s^®/Dockers^® business. The Company also announced that it would exit its Candies^® outlet business, which it did by the end of fiscal 2002. These decisions enabled management to focus its resources on growing its more profitable Casual Male business and, to a lesser extent, expanding the operations of its Ecko joint venture. See “Other Branded Apparel Businesses-Levi’s^®/Dockers^® and Candies^® outlet businesses” for a description of the Company’s exit strategy.

In June 2003, the Company announced an exclusive marketing agreement with George Foreman to promote the Casual Male brand. In March 2004, the Company launched its national marketing campaign introducing its exclusive line of clothing bearing the George Foreman name. The clothing, which will be marketed under three product lines, “Comfort Zone By George Foreman^™,” “Signature Collection By George Foreman^™” and “GF Sport By George Foreman^™,” will include a complete array of casual wear, dresswear and accessories and will include the comfort zone technology that the Company previously sold under its private label brand.

Since the Casual Male acquisition, the Company has operated in two segments, its “Casual Male business” and its “Other Branded Apparel businesses.”

CASUAL MALE BUSINESS

The Casual Male business, which represents over 74% of the Company’s total revenues for fiscal 2003, is a multi-channel retailer offering what the Company believes to be high quality casual wear for the big and tall customer. The Company offers its merchandise to customers through diverse selling and marketing channels, including over 481 retail and outlet stores, which operate under the names “Casual Male Big & Tall^®” and “Casual Male Big & Tall Outlet” stores, its Casual Male Big & Tall catalog and two e-commerce sites.

Industry

The NPD Group, a leading marketing information provider, estimates that the men’s big and tall apparel market, which includes pants with a waist size 44” and greater, as well as tops sized 1X and greater, generated approximately $5.3 billion in sales in 2002. This highly fragmented market represents approximately 13% of the overall men’s apparel business. In 2002, sales of men’s big and tall apparel increased 1.4%, while overall men’s apparel sales declined 0.7%. Growth in this segment has been driven by rapidly changing market demographics. Currently, 64% of U.S. adults are overweight or obese, up from 56% six years ago. Additionally, in 2001, 49 states classified 15% or more of their total adult population as obese, versus four states in 1991. Moreover, 29 states classified 20% or more of their total adult population as obese. According to the Center for Disease Control, the rate of obesity for the under-30 age group is growing faster than any other segment of the population. These statistics suggest that there is a significant gap between the market share of the big and tall apparel market and the overall percentage of the population classified as obese.

The men’s big and tall apparel market is currently served by a variety of retailers, including department stores, mass merchandisers and specialty stores. These stores typically offer a limited assortment of sizes and styles. By offering the widest selection of sizes and styles, the specialty channel has gained significant market share over the past five years, accounting for 26.2% of men’s big and tall apparel sales in 2002 versus 22.7% in 1998. Casual Male’s overall market share of the men’s big and tall apparel market approximates 7%.

4

Table of Contents

Multi-Channel Strategy

The Company tries to appeal to all of its customers by providing multiple ways to shop. The Casual Male customer is often a destination shopper when it comes to purchasing apparel for himself. For customers that are somewhat uncomfortable in a traditional shopping environment, the Company offers them the opportunity to shop through its catalog as well as through the Company’s e-commerce sites. The Company has recently begun to coordinate its merchandising and marketing across its stores, catalog and websites.

Retail stores. At January 31, 2004, the Company operated 415 Casual Male Big & Tall full-price retail stores, located primarily in strip centers, power centers or stand-alone locations. These stores target the middle-income customer seeking good quality at moderate prices. These stores offer a broad selection of basic sportswear and other casual apparel, as well as dress wear and accessories. The average Casual Male store is approximately 3,400 square feet and has approximately $178 in sales per square foot annually. The Company believes that it has a significant opportunity to grow its retail stores base, and has identified more than 250 potential additional locations.

Outlet stores. At January 31, 2004, the Company operated 66 Casual Male Big & Tall outlet stores designed to offer a wide range of casual clothing for the big and tall customer at prices that are generally 20-25% lower than those offered at it retail stores. Most of the merchandise in the Company’s outlet stores is offered with the purchasing interests of the value-oriented customer in mind. Approximately 20% of the merchandise in the outlet stores originates from the retail stores and serves as a distribution channel for clearance and other slow moving inventory. The average Casual Male outlet store is approximately 3,100 square feet and has approximately $172 in sales per square foot annually. The Company believes that there are an additional 50 potential locations suitable for a Casual Male outlet store.

For fiscal 2004, the Company intends to open 15 new stores, which will be a combination of retail and outlet stores. In addition, the Company expects to remodel between 300 to 350 of its existing retail and outlet stores over a two year period beginning in fiscal 2004. Many of these stores are aging and have not been updated since they opened. The remodel program will include replacing carpeting, repainting the store and otherwise repairing parts of the store, where needed. In addition, certain stores require updating fixtures and signage where necessary to maintain consistent presentation standards. In total, the average cost of the remodel program will approximate between $35,000 to $40,000 for each store location and will require approximately two years to complete such program.

Catalog. The Company’s “Casual Male Big & Tall” catalog offers an assortment of casual merchandise similar to what is available in the stores, but also offers a broader selection of dress apparel, such as sportcoats, suit separates and other tailored clothing. The Company issued 17 editions of its catalog in fiscal 2003 and circulated a total of 7.5 million catalogs. In fiscal 2004, the Company intends to enhance the productivity of its catalog by reducing circulation by approximately 10% and focusing its mailings on a more targeted group, based on historical buying patterns.

In the second half of fiscal 2003, the Company started to transition its REPP Big & Tall catalog customers to its Casual Male Big & Tall catalog. The REPP Big & Tall catalog was an extension of the REPP Ltd. Big & Tall stores, which were previously owned and subsequently closed by Casual Male Corp., prior to the Company’s acquisition.

In-bound calls and order fulfillment for the Company’s catalog are currently handled at its distribution center. See “Distribution” below.

E-Commerce. The Company currently has two e-commerce web sites: www.casualmale.com and www.reppbigandtall.com. These sites offer substantially the same merchandise as is offered in its catalog. The Company currently processes and fulfills orders from its web sites through its distribution center. Although its e-commerce web sites have generated relatively small sales volume, the Company sees the internet as a significant growth opportunity. For example, in addition to its traditional on-line business, the Company recently launched Casual Male Big & Tall apparel shops on Amazon.com. The Company expects its relationship with Amazon.com will provide further growth and brand identity for its Casual Male business. During fiscal 2003, the Company’s e-commerce business sales increased by 49%.

Similar to the REPP Big & Tall catalog, during fiscal 2004, the Company will begin to phase out its Repp Big and Tall web site and combine it with its existing www.casualmale.com web site.

5

Table of Contents

Merchandise

The Casual Male business offers an extensive selection of quality sportswear, dress clothing and footwear for the big and tall male customer at moderate prices. Over 50% of the Casual Male merchandise is basic or fashion-neutral items, such as jeans, casual slacks, tee-shirts polo shirts and dress shirts. The Casual Male customer is primarily interested in comfort and fit, more so than price. As such, Casual Male’s clothing has features specifically designed for our customer, such as waist-relaxer pants, stretch belts, zipper ties, wide band socks, neck-relaxer shirts and clothing with comfort-stretch technology and reinforced stress points. Independent tests have shown that the Company’s merchandise is among the highest in quality of all major brands. In fiscal 2003, approximately 75% of the merchandise offered was private label, manufactured primarily under its Harbor Bay and Menswear brand names. The Company expects the percentage of these private labels to decrease in fiscal 2004.

During Spring, 2004, the Company launched its new line of men’s big and tall apparel featuring Comfort Zone by George Foreman^™, Signature Collection by George Foreman^™ and GF Sport by George Foreman^™ in its stores, catalog and on its Casual Male e-commerce site.

George Foreman branded apparel is expected to approximate 15% of the Casual Male merchandise assortment in Spring 2004 and approximately 30% of the Fall 2004 assortment. The Company intends to continue exploring other assortment additions to the George Foreman apparel line depending on performance and adaptation of existing assortments to include comfort zone features.

In addition, the Company has successfully negotiated with manufacturers of several well-known brands and expects that this branded merchandise, which is expected to have merchandise margins similar to Harbor Bay, could represent up to 60% of the Company’s merchandise mix in fiscal 2004. The Company already carries a wide range of well-known brand name merchandise including, Ecko Unltd.^®, Polo Jeans Co.^®, Perry Ellis Portfolio^®, Geoffrey Beene^®, Sean John^®, Sketchers^®, Levi’s^®, Dockers^® and others. In addition, Casual Male will launch an exclusive Reebok^® big and tall collection for Fall 2004. Accordingly, it is anticipated that Casual Male’s private label Harbor Bay and Menswear merchandise assortments will be displaced significantly by the George Foreman lines of apparel, or other branded merchandise assortments.

The Company recently completed a roll-out of its new lifestyle format in all of its Casual Male Big and Tall stores. Stores are now merchandised to showcase entire outfits by lifestyle, including traditional, functional active, young men’s, dresswear and contemporary. This format allows the Company to merchandise key items and seasonal goods in prominent displays, makes coordinating outfits easier for the customer and encourages multi-item purchases. This lifestyle layout also allows the Company to better manage store space in each market to target local demographics. Stores are clustered to meet

6

Table of Contents

the demographic needs of customers by climate and ethnicity. The key item strategy is also fully integrated by lifestyle, allowing the Company to focus on merchandise presentation and offer its customers a compelling value proposition. By taking advantage of volume purchasing discounts, the Company is able to promote these key items across the entire chain without sacrificing gross margins.

The Company’s initiative to add extended sizing in tall sizes in core programs for Spring 2004 has resulted in doubling the penetration of these sizes to the total. Casual Male intends to offer additional extended sizes in core programs for Fall 2004.

To improve upon the Company’s capabilities to plan and replenish its merchandise assortments according to the age and demographic characteristics of each market, Casual Male adopted the Arthur product planning methodologies in 2004, as well as installed and implemented the E3 replenishment application. Both of these applications are part of the JDA Enterprise Portfolio. These new applications will not only improve upon Casual Male’s merchandise planning capabilities, but will also enable it to better manage size requirements for each store. Subsequent to implementing the JDA Enterprise Portfolio, intended to be live by mid-2004, Casual Male plans to pursue more advanced assortment planning tools to further enhance its abilities to provide merchandise assortments tailored to the individual characteristics of each market.

Marketing and Advertising

The Company’s marketing department creates and implements a wide variety of national, regional and local advertising, direct marketing and sales promotion programs. These television, radio, direct mail and e-mail programs are designed to increase sales and customer awareness of the Casual Male brand name. Approximately 70% of all sales are captured in the Company’s customer database. This allows continuous communication with the Company’s customers both for promotional events as well as relationship programs such as the birthday club and new customer programs. Local store marketing activities occur on a regular basis and include store opening events and in-store promotion programs. For 2004, the Company has redesigned its marketing strategy with the objective of branding Casual Male and acquiring new customers.

Advertising and marketing costs for the Casual Male businesses represented approximately 5.8% of the revenue from the Casual Male business in fiscal 2003. This includes creating and distributing 7.5 million catalogs, over 22 million retail direct mail pieces and all in-store signage for its 481 stores, as well as supporting e-commerce efforts. Historically through the end of fiscal 2003, approximately 66% of the Company’s total marketing budget was spent on direct marketing, utilizing its customer database which contains more than 2.5 million names, together with their respective Casual Male purchase histories. In 2004, the direct mail marketing programs percentage of total budget will be decreased to approximately 30% and includes 17 million contacts in support of 19 promotional events.

In fiscal 2004, as part of its new strategy the Company plans to redirect a portion of its direct marketing dollars in addition to increasing the total advertising budget to include for the first time outreach marketing initiatives, such as television. The Company believes that this type of marketing is necessary in order to attract new customers and introduce them to the Casual Male brands, including its George Foreman brand product lines. Over the past several years, the direct marketing program was geared toward existing customers. Although effective with the current customer base, a direct marketing program does not support the goal of increasing market share. The Company expects that its overall advertising budget will be approximately 7.5% of sales in fiscal 2004, an increase from approximately 5.8% in fiscal 2003, of which approximately 50% of the marketing budget will represent outreach marketing programs, mainly television. This will include the launch of a broad-range advertising campaign which commenced in March 2004 in connection with the Company’s roll out of the new George Foreman line of merchandise. Both national network and cable television will run throughout the year with the specific goals of branding Casual Male and customer acquisition in all three channels of distribution: retail stores, catalog and e-commerce.

With the celebrity status of George Foreman associated with Casual Male, a key component of marketing both the brand Casual Male and the new exclusive George Foreman line of apparel is public relations. Internal and external resources have been allocated to support this effort. This program started in the fall of 2003 and will continue through 2004 with media tours and press releases of new product during key selling periods.

7

Table of Contents

In Fall 2004, coupled with store systems upgrades, the development of a new customer relationship management system will be put in place. This will allow the Company to expand its level of communication outside of direct mail, and pertinent customer information will be available at the store level, with the goal of servicing the customer better. This program is expected to be rolled out in early 2005 and will include a loyalty program.

Competition

The Casual Male business faces competition from a variety of sources, including department stores, specialty stores, discount and off-price retailers, as well as other retailers that sell big and tall merchandise. While the Company has successfully competed on the basis of merchandise selection, favorable pricing, customer service and desirable store locations, there can be no assurances that other retailers will not adopt purchasing and marketing concepts similar to those of the Company. In addition, discount retailers with significant buying power, such as Wal-Mart, J.C. Penny and Target, represent a source of competition for the Company.

The United States men’s big and tall apparel market is highly competitive with many national and regional department stores, specialty apparel retailers, single market operators and discount stores offering a broad range of apparel products similar to those of the Company. Besides retail competitors, the Company considers any casual apparel manufacturer operating in outlet malls throughout the United States to be a competitor in the casual apparel market. The Company believes that it is the only national operator of apparel stores focused on the men’s big and tall market, the next largest specialty retailer being Rochester Big and Tall, which operates 22 stores.

The catalog business has several competitors, including the King Size Catalog (which is owned by Brylane) and J.C. Penny’s Big and Tall Collection catalog.

DISTRIBUTION

The Company acquired its property in Canton, Massachusetts, which represents its corporate offices and distribution center, as part of the Company’s acquisition of the Casual Male business. At the end of fiscal 2002, the Company terminated the leases of its two other distribution centers, which were located in Orlando, Florida, and centralized all of its distribution operations in Canton.

The Company believes that having one centralized distribution facility minimizes the delivered cost of merchandise and maximizes the in-stock position of its stores. The Company believes that the centralized distribution system enables its stores to maximize selling space by reducing necessary levels of back-room stock carried in each store. In addition, the distribution center provides order fulfillment services for the Company’s e-commerce and catalog businesses.

In fiscal 2003, the Company partnered with United Parcel Services (“UPS”) to improve upon its distribution methods and reduce shipping costs as a result of not having to use third party trucking companies. By utilizing UPS, the Company is able to track all deliveries from the warehouse to the individual stores, as well as the status of in-transit shipments.

In fiscal 2004, the Company expects to implement Manhattan Associates’ PKMS warehousing application for its distribution center systems, which is expected to significantly streamline its distribution processes, enhance in-transit times, and significantly reduce distribution costs. In addition, Casual Male has slightly altered its distribution strategy as it relates to seasonal merchandise to allow for replenishment of styles at the color and size level which is intended to optimize sales opportunities and minimize end of season clearance markdowns. This seasonal merchandise distribution strategy requires significantly higher volume of individual piece processing, a more costly distribution method in contrast to carton handling. However, in conjunction with the installation of a more efficient PKMS warehousing application, the Company is also updating its processes in order to optimize the new system capabilities in areas such as receiving, cross-dock handling, put-away, and picking which will include the use of scanning equipment. Accordingly, in spite of the shift in distribution strategy to include the more costly piece replenishment, Casual Male’s distribution costs are expected to decrease with the installation of new systems and the adoption of updated process flows.

8

Table of Contents

MANAGEMENT INFORMATION SYSTEMS

Since the Company’s acquisition of the Casual Male business, one of the Company’s primary goals has been to integrate the systems of the acquired business with those of the Company and to upgrade and enhance current systems where necessary. The Company has identified significant cost savings and synergies that can only be realized upon the completion of this integration and upgrade of the system infrastructure.

At the time of the acquisition, the Casual Male business operated primarily on an IBM mainframe platform. The mainframe based system included an internally supported enterprise retail system and a human resource/payroll application. The Company’s then-existing e-commerce/catalog fulfillment infrastructure operated on a Hewlett-Packard environment on software from Ecometry, and the remainder of is business operated on an IBM 400 platform. The Company’s financial systems were supported by a client server environment.

Presently, the Company’s management information systems consist of a full range of retail merchandising and financial systems which include merchandise planning and reporting, distribution center processing, inventory allocation, sales reporting, and financial processing and reporting. Until integration is complete, the Company’s Casual Male business will continue to operate primarily on a mainframe platform, with the e-commerce/catalog business on the HP environment, and the remainder of the business on the IBM AS/400 platform.

In February 2003, the Company completed the conversion of the Casual Male business to its Lawson Financial System. Management now has several tools from which to manage the business on a consolidated level in a more efficient and effective manner. During fiscal 2003, the Company implemented the JDA E3 Advanced Replenishment system for the Casual Male business. This system was integrated with the existing Casual Male business mainframe platform in an effort to improve sales opportunities and better manage inventories of basic merchandise.

During fiscal 2004, the Company plans to complete the systems integration work to upgrade its merchandising systems to the JDA Portfolio Solutions, specifically the MMS Merchandise Management System, E3 Advanced Replenishment, Retail Ideas Data Warehouse, and Arthur Merchandise Planning and Advanced Allocation systems. In addition, the Company also plans to convert its distribution system to Manhattan Associates’ PKMS distribution system.

To implement these initiatives, the Company spent approximately $4.1 million in fiscal 2002 and $4.0 million in fiscal 2003 and expects to spend approximately $6.0 million in fiscal 2004, primarily on system enhancement, implementation and maintenance. Once the systems are fully implemented, the Company expects to be able to improve sales, better manage inventory levels and streamline operations by:

Currently, all of the Company’s stores have point-of-sale terminals supplied by IBM, Fujitsu, and NCR. The Casual Male business is supported by a point-of-sale business application provided by Triversity, and the remainder of the Company’s business is supported by a point-of-sale (“POS”) business application by CRS. The POS applications capture daily transaction information by item, color and size. The Company utilizes barcode technology in tracking sales, inventory and pricing information. Communication between the corporate headquarters and all stores is facilitated on a daily basis through the use of an electronic mail system.

In fiscal 2005, the Company anticipates upgrading to a POS and register systems in its store locations. This new system will include a multitude of features including Customer Relationship Management tools that will enable the Company to track customer buying habits and provide it with the ability to target customers with specific offers and promotions. The Company expects that this will also enable the Company to develop an effective loyalty program for its customers. In March 2004, the Company announced that it signed a license agreement with NSB Group for its Connected Retailer^® Store and Customer Relationship Management Solutions. The Company expects the roll-out of this project to its stores will begin in Spring 2005. The Company believes that these efforts can lead to new customers as well as a more loyal

9

Table of Contents

customer base. The cost of this project is anticipated to approximate $6.0 million, including a five-year maintenance arrangement on its store registers.

The merchandising management systems are updated daily with all store transactions and provide daily sales, inventory, pricing and merchandise information and management reports to assist the Company in operating its retail business. The Company’s merchandising system applications also facilitate the placement and tracking of purchase orders and utilize EDI technology. The Company evaluates this information, together with weekly reports on merchandise statistics, prior to making decisions regarding reorders of fast-selling items and the allocation of merchandise.

The Company utilizes a Microsoft NT / Windows 2000 environment running on a local area network to communicate and work-share within its corporate headquarters. The Company also utilizes the services of ADP, an outside payroll processing provider, to prepare, distribute and report its weekly payroll. All of the Company’s payroll processing was consolidated onto the ADP processing system effective January 1, 2003.

OTHER BRANDED APPAREL BUSINESSES

The Ecko Unltd.^® Business

The Company entered into a joint venture arrangement with Ecko, under which the Company owns and manages retail outlet stores bearing the name Ecko Unltd.^® and featuring Ecko^® branded merchandise. The Company believes that the Ecko Unltd.^® outlet stores represent an opportunity in the outlet marketplace for the underdeveloped young men’s and junior market because Ecko^® is believed to be a cross-over youth brand appealing to both urban and suburban youth with a core customer between the ages of 14 to 24. Accordingly, these stores provide a broad selection of merchandise ranging from hip-hop to extreme sports, and street-wear to fraternity wear, with a core menswear line consisting of fleece, twill and denim bottoms, wovens, printed tee shirts, knits and sweaters. The average Ecko Unltd.^® outlet store is approximately 3,800 sq. ft. and generates an average of $311 sales per square foot annually. At January 31, 2004 the Company operated 20 Ecko Unltd.^® outlet stores. The Company intends to open 10 new Ecko stores in fiscal 2004 and will be increasing the size of these stores to 4,000 square feet. In November 2003, the Company opened its first mall-based, full-priced Ecko Unltd.^® retail store.

Management believes that its Ecko Unltd.^® outlet stores compete with other outlet apparel retailers on the basis of selection of quality, service and price. The Company stresses product training of sales staff and, with the assistance of merchandise materials from Ecko, provides the sales personnel with substantial product knowledge and training across all product lines.

The Company owns 50.5% of this joint venture and Ecko owns the remaining 49.5% of the joint venture. Under the terms of the joint venture arrangement, the Company manages the Ecko Unltd.^® retail outlet stores and Ecko contributes to the joint venture their trademark and merchandise requirements at cost. Further, the Company contributes all real estate and operating requirements for the retail outlet stores, including, but not limited to, the real estate leases, payroll needs and advertising. Each partner shares in the operating profits of the joint venture, after each partner has received reimbursement for its cost contributions. Pursuant to its arrangement, the Company must maintain a prescribed store opening schedule and open 75 stores over a six-year period in order to maintain the joint venture’s exclusivity. At certain times during the term of the agreement, the Company may exercise an option to sell its share of the joint venture to Ecko, and Ecko has an option to acquire the Company’s share of the joint venture at a price based on the performance of the Ecko retail outlet stores.

Levi’s^®/Dockers^® and Candies^® outlet businesses

The Company announced in fiscal 2002 that it would be exiting its Levi’s^®/Dockers^® outlet business. At January 31, 2004, the Company operated 58 Levi’s^®/Dockers^® outlet stores, which it still intends to close over the next two years. The Company expects that these remaining 58 stores will either be closed at the end of their respective lease terms or through negotiations with respective landlords. Until such closings, these remaining stores will continue to offer an exclusive selection of Levi Strauss & Co. brands of merchandise, which include Levi’s^® and Dockers^® brands, at outlet prices. As such, the Company will continue to work with Levi Strauss & Co. on obtaining competitive costs for its products and, based on availability, will stock these remaining stores with a higher level of close-out merchandise, which will enable the

10

Table of Contents

Company to maintain its margin levels. The Company expects that these remaining stores will operate at or above break-even levels.

LP INNOVATIONS, INC.

LP Innovations, Inc. (“LPI”), an 80% owned subsidiary of the Company, which was acquired by the Company in May 2002 as part of its Casual Male acquisition, is a national provider of loss prevention solutions to companies in the retail industry. LPI’s loss prevention services include in-store audits, employee investigations, employee training and awareness, and specially designed software. Through LPI’s subsidiary, Securex LLC, LPI also sells and installs security equipment and alarm and security central monitoring, allowing LPI to provide a comprehensive loss prevention solution to its clients.

LPI’s clients have historically included small, mid-sized and large retail chains in the specialty, grocery, apparel, office supplies, sporting goods and numerous other segments of the retail industry. LPI currently has approximately 50 loss prevention retail clients representing over 5,000 retail locations. Approximately 64.8% of these retail clients have contracted for LPI’s full outsource loss prevention services, while the balance of the retail clients have contracted for a portion of LPI’s services, such as in-store audits or in-store employee investigation. For fiscal year 2003, loss prevention services generated revenues of approximately $6.2 million, which accounted for approximately 64.5% of LPI’s total revenues of approximately $9.5 million and approximately $698,000, or 22.6%, of the total loss of $3.1 million. Included in the total loss prevention revenues of $9.5 million are revenues of $1.7 million generated by services provided to the Company by LPI, or 18.0% of total loss prevention revenues.

From a background perspective, LPI was started in May 1988 to provide loss prevention services to its then parent, Casual Male Corp. In 1999, LPI began to experiment with an outsourcing model by providing loss prevention services to other retail companies. In January 2001, LPI expanded beyond loss prevention services to include providing security system sales and installation and security central monitoring services by acquiring Securex LLC. During July of fiscal year 2002, LPI made the strategic decision to realign its Securex-related business as more of a support function of its core loss prevention services that would then be able to generate revenues organically through LPI’s existing customer base. With the main focus on its outsourced loss prevention services and continued client and revenue growth, LPI intends to more efficiently utilize the nationwide loss prevention service infrastructure it has created. Although LPI has made and continues to make changes to its business to facilitate the realization of its business plan, there can be no assurance that LPI will be successful in expanding its retail client base and capitalizing upon its established infrastructure.

On February 23, 2004, subsequent to the end of fiscal 2003, LP Innovations, Inc. (“LPI”) filed a Form 10 registration statement with the Securities and Exchange Commission in connection with a proposed pro rata distribution of the Company’s 80% ownership interest in LPI to the Company’s existing stockholders. Under the proposed plan of distribution, each holder of the Company’s Common Stock will receive one share of LPI common stock for every 100 shares of the Company’s Common Stock owned at the date of distribution. Immediately after the distribution, which is expected to occur in the second quarter of fiscal 2004, LPI will be a separate and independent public company. After the distribution is complete, LPI’s board of directors is considering, among other financing alternatives, offering its stockholders the right to purchase additional shares of LPI common stock through a rights offering to raise additional funds for working capital and to repay amounts owed to the Company, which approximated $5.0 million at January 31, 2004. For the fiscal year ended January 31, 2004, the Company wrote off approximately $355,000 of costs incurred in connection with LPI becoming a stand alone entity. The proposed distribution of LPI common stock is currently being reviewed by the SEC as part of the normal registration process. The timing and terms of the proposed distribution may be affected by this process.

SEASONALITY

Consistent with the retail industry, the Company’s business is seasonal. The Casual Male business traditionally generates the largest volume of its sales during the Father’s Day selling season in June and the Christmas selling season in December. The Company’s outlet businesses, which include its Levi’s^®/Dockers^® and Ecko Unltd.^® outlet stores, traditionally generate the largest volume volumes during the back-to-school selling season in August and the Christmas selling season in December. The majority of the Company’s operating income is generated in the fourth quarter as a result of the impact of the Christmas selling season.

11

Table of Contents

TRADEMARKS/TRADEMARK LICENSE AGREEMENTS

The Company owns several servicemarks and trademarks relating to its Casual Male business, including Casual Male^® and “Casual Male Big & Tall^®”, “Harbor Bay^®”, “GF Sport By George Foreman^™”, “Comfort Zone by George Foreman^™”, “George Foreman Signature Collection^™” and “Signature Collection By George Foreman^™” The Company also has a U.S. patent pending for an extendable collar system, which is marketed as “Neck Relaxer^™.”

The Company operates its Levi’s^®/Dockers^® business pursuant to a trademark license agreement with Levi Strauss & Co., which agreement was most recently amended in October 1998 (as amended, the “Levi Outlet License Agreement”). The Levi Outlet License Agreement authorizes the Company to use certain Levi Strauss & Co. trademarks in connection with the operation of the Company’s Levi’s^®/Dockers^® Outlet by Designs in 25 states in the eastern portion of the United States and in Puerto Rico.

The Company operates its Ecko Unltd.^® outlet stores pursuant to a joint venture arrangement with Ecko.Complex, LLC which authorizes the Company to utilize the Ecko^® trademark in connection with the operations of its Ecko Unltd.^® outlet stores.

EMPLOYEES

As of January 31, 2004, the Company employed approximately 3,916 associates, of whom 2,553 were full-time personnel. The Company hires additional temporary employees during the peak Fall and Holiday seasons. None of the Company’s employees is represented by any collective bargaining agreement.

AVAILABLE INFORMATION

The Company’s corporate web site is www.cmrginc.com. The Company makes available, free of charge, through its web site its Annual Reports on Form 10-K, Quarterly Reports on Form 10-Q, Current Reports on Form 8-K and amendments to such reports filed or furnished pursuant to Section 13(a) of the Securities Exchange Act of 1934, as amended, as soon as reasonably practicable after the Company has electronically filed such material with, or furnished such materials to, the Securities and Exchange Commission.

12

Table of Contents

The Company’s corporate offices and distribution center are located at 555 Turnpike Street in Canton, Massachusetts. The facility is located on 37 acres of land and is owned by Designs Canton Property Corp., a wholly owned subsidiary of Designs Canton Holdings, Inc., which is an indirect wholly owned consolidated subsidiary of the Company. The property, which was acquired by the Company as part of its Casual Male acquisition in May 2002, is subject to an outstanding mortgage of $10.7 million at January 31, 2004. The mortgage is secured by the real estate and the buildings. The building contains about 750,000 square feet, which includes approximately 150,000 square feet of office space.

The Company remains liable on its lease, which expires in January 2006, for its previous corporate offices located at 66 B Street, Needham, Massachusetts. In January 2003, the Company entered into a sublease agreement with its loss prevention subsidiary, LP Innovations, Inc. (“LPI”), pursuant to which LPI leases approximately 19,000 square feet of the total 80,000 square feet of this facility. The sublease agreement is for three years and will expire on December 31, 2005.

As of January 31, 2004, the Company operated 415 Casual Male Big and Tall retail stores, 66 Casual Male Big and Tall outlet stores, 58 Levi’s^®/Dockers^® Outlet stores and 21 Ecko Unltd.^® outlet and retail stores. All of these stores are leased by the Company directly from shopping center owners. The store leases are generally five years in length and contain renewal options extending their terms to between 5 and 10 years. Following this discussion is a listing by state of all the Casual Male Big and Tall retail and outlet store locations open at January 31, 2004.

Sites for store expansion are selected on the basis of several factors intended to maximize the exposure of each store to the Company’s target customers. These factors include the demographic profile of the area in which the site is located, the types of stores and other retailers in the area, the location of the store within the center and the attractiveness of the store layout. The Company also utilizes financial models to project the profitability of each location using assumptions such as the center’s sales per square foot averages, estimated occupancy costs and return on investment requirements. The Company believes that its selection of locations enables the Company’s stores to attract customers from the general shopping traffic and to generate its own customers from surrounding areas.

See also “Management’s Discussion and Analysis of Financial Condition and Results of Operations—Liquidity and Capital Resources—Capital Expenditures.”

13

Table of Contents

Store locations by State for the Casual Male Big and Tall Retail and Outlet stores at January 31, 2004

14

Table of Contents

Store locations by State for the Casual Male Big and Tall Retail and Outlet stores at January 31, 2004

15

Table of Contents

On October 15, 2003, a class action lawsuit was filed against the Company in the Supreme Court of California, County of Santa Clara, by Robin J. Tucker, a former employee. The complaint alleges, among other things, that the Company failed to pay overtime compensation and to provide meal and rest periods to the Company’s California store managers for the period from May 14, 2002 to the present. The Company believes that it has substantial legal defenses to these claims and intends to vigorously defend this action. However, there can be no assurances that such claims will not be successful in whole or in part.

On April 2, 2004 a former officer of the Company filed a discrimination charge with the Massachusetts Commission Against Discrimination and the Equal Employment Opportunity Commission. The Company will engage an outside independent fact-finder to conduct a thorough investigation into the allegations. The claim has been submitted to the Company’s insurance carrier for indemnity and defense. The Company does not expect this matter to have a material impact on its financial position or results of operation.

No matters were submitted to a vote of the Company’s security holders during the fourth quarter of fiscal 2003.

16

Table of Contents

PART II.

The Company’s Common Stock is listed for trading on the Nasdaq National Market under the symbol “CMRG.”

The following table sets forth, for the periods indicated, the high and low per share closing sales prices for the Common Stock, as reported on the Nasdaq consolidated reporting system.

As of April 10, 2004, based upon data provided by independent shareholder communication services and the transfer agent for the Common Stock, there were approximately 306 holders of record of Common Stock.

The Company has not paid and does not anticipate paying cash dividends on its Common Stock. In addition, financial covenants in the Company’s loan agreement may restrict dividend payments. For a description of these financial covenants see Note C to the Notes to the Consolidated Financial Statements.

On November 17, 2003, the Company issued $85.0 million principal amount of convertible senior subordinated notes due 2024 (the “Convertible Notes”). On November 26, 2003, the Company issued an additional $15.0 million of Convertible Notes, which were sold pursuant to an option exercised by the initial purchaser. The aggregate $100 million principal amount of Convertible Notes bear interest at a rate of 5% per year and are convertible into the Company’s Common Stock at a conversion price of $10.65 per share. These Convertible Notes were issued in reliance upon the exemptions from the registration requirements of the Securities Act of 1933, as amended (the “Securities Act”), set forth under Section 4(2) of the Securities Act or Regulation D promulgated thereunder, relative to sales by an issuer not involving any public offering.

In connection with the Company’s private placement of $29.6 million of senior subordinated notes due 2010 which were issued during the second and third quarters of fiscal 2003, the Company issued detachable warrants to purchase 1,182,400 shares of the Company’s Common Stock at exercise prices ranging from $4.76 to $7.32 per share. The Company assigned a fair value to these warrants of $5.6 million. These warrants were issued in reliance upon the exemptions from the registration requirements of the Securities Act of 1933, set forth under Section 4(2) of the Securities Act or Regulation D promulgated thereunder, relative to sales by an issuer not involving any public offering.

See Note C to the Notes to the Consolidated Financial Statements for a complete description of these private placements.

17

Table of Contents

Stock Repurchases

The following table includes repurchases by the Company of its Common Stock during the fourth quarter of fiscal 2003:

18

Table of Contents

Selected Financial Data-Historical Results

19

Table of Contents

Selected 5 Year Financial Data of Casual Male

The following table provides historical financial information of the Casual Male business for the past five fiscal years, of which all information prior to May 14, 2002 is according to the financial records of Casual Male Corp., the predecessor company

20

Table of Contents

FORWARD LOOKING STATEMENTS

As noted above, this Annual Report on Form 10-K, including, without limitation, this Item 7, contains “forward-looking statements” within the meaning of the United States Private Securities Litigation Reform Act of 1995. Actual results or developments could differ materially from those projected in such statements as a result of numerous factors, including, without limitation those risks and uncertainties set forth in the Company’s Current Report on Form 8-K, previously filed with the SEC on April 14, 2004, which you are encouraged to read. The following discussion and analysis of our financial condition and results of operations should be read in light of those risks and uncertainties and in conjunction with our accompanying Consolidated Financial Statements and Notes thereto.

EXECUTIVE OVERVIEW

Since acquiring the Casual Male business from Casual Male Corp. in May 2002, the Company has been working on a three-year turnaround plan aimed toward:

Over the past twenty-three months many of these objectives have either been accomplished or are in the latter stages of completion, positioning the Company to benefit from these changes over the coming quarters. Since acquiring the Casual Male business, the Company has successfully reduced its selling, general and administrative by approximately 15% on an average store basis and formulated and commenced implementation of an enterprise wide systems strategy. One of the most significant changes has been an updating of Casual Male’s merchandising strategy. During fiscal 2003, the Company implemented numerous merchandising initiatives to more broadly appeal to its varied customer base such as the introduction of its young men’s line of apparel, the re-introduction of suit separates, the inclusion of more branded merchandise in its assortments, including Comfort Zone by George Foreman, Signature Collection by George Foreman and GF Sport by George Foreman, the proliferation of its key item merchandising program, and the extension of greater range of sizes available in assortments. These merchandising initiatives, which started to be implemented in the second quarter of fiscal 2003, have resulted in a marked improvement in quarter to quarter comparable store sales. In the first quarter of fiscal 2003 comparable store sales were a negative 5% as compared to the corresponding period of the prior year, improving each quarter to a positive 2.4% in the fourth quarter. Although it is still early in the Company’s first quarter of fiscal 2004, this trend of improvement appears to be continuing into next year.

In addition to its merchandising strategy, the Company is also focused on improving the customer traffic to its stores. One significant statistic the Company uses to understand its sales performance is its conversion rate, which is the percentage of total customers who enter our stores that purchase merchandise. The Company’s conversion rates in fiscal 2003 have improved by approximately 10% over fiscal 2002. However, customer traffic to its stores continues to be down over 10% as compared to the prior year. The Company believes that an improved marketing strategy can reverse the negative traffic trend. In the past, much of the marketing strategy has been geared toward its existing customer. Toward the end of fiscal 2003, the Company revamped its marketing program and started to redirect its objective to re-introducing the Casual Male brand and its multi-channel businesses. The Company plans to accomplish this primarily through increased media, television and radio. Some of this increased marketing spend contributed to the slight increase in selling, general and administrative expenses incurred in fiscal 2003.

To help further promote the Casual Male concept, in June 2003, the Company signed George Foreman as the Company’s spokesperson and also developed three lines of product bearing the George Foreman name. Although the Company’s

21

Table of Contents

national marketing launch will not occur until Spring 2004, the sales performance of the George Foreman product lines for the fourth quarter of fiscal 2003 were positive.

From a financial position perspective, the Company completed the sale of $100 million in 5% convertible notes during the fourth quarter of fiscal 2003. This offering enabled the Company to substantially prepay all of its high rate debt, much of which was acquired in conjunction with the Casual Male acquisition, and pay off amounts outstanding under its credit facility. In addition to saving the Company approximately $4.0 million in annual interest costs, this offering also increased the Company’s liquidity for working capital needs. In connection with the Company’s prepayment of its debt, the Company incurred approximately $13.8 million in costs, primarily associated with the write-off of deferred costs. See the full discussion of the repayments of debt and related charges below under “Liquidity and Capital Resources.” The Company sees this offering as a significant step toward a stronger and more liquid balance sheet while at the same time supporting the Company’s future growth strategies.

SEGMENT REPORTING

Since the Casual Male acquisition, the Company has redefined its business into two reportable business segments: (i) the Casual Male business and (ii) Other Branded Apparel businesses. See Note M of the Notes to Consolidated Financial Statements for a complete disclosure of the financial results of each segment. The Company’s “Other Branded Apparel businesses” segment includes the operations of the Company’s Levi’s^®/Dockers^® outlet stores and its Ecko Unltd.^® outlet and retail stores.

STORE CLOSINGS/DISCONTINUED OPERATIONS

In accordance with the provisions of Statement of Financial Accounting Standard (“SFAS”) No. 144, Accounting for the Impairment or Disposal of Long-Lived Assets (“SFAS No. 144”), results of operations for stores closed in fiscal 2003 and fiscal 2002 have been presented as discontinued operations. The discontinued operations of these stores include related write-downs for the impairment of assets and costs to liquidate inventory. Accordingly, the Company has reclassified all prior year results for such closed stores to conform with the current period presentation of discontinued operations.

RESULTS OF OPERATIONS

Because of the materiality of the Casual Male business to the Company’s consolidated operations, the following discussion of results of operations will first review the segment results of the Casual Male business and will conclude with a consolidated review of the Company’s results. The discussion of the Company’s consolidated results of operations begins on page 25 of this Form 10-K. Because the actual results for the Casual Male business include only operations from May 14, 2002, the date of acquisition, the following discussion will also include pro forma results from operations for fiscal 2002 and fiscal 2001, assuming that the acquisition of Casual Male occurred on February 4, 2001. Management believes that this additional information is necessary in order to provide a complete and balanced discussion of the results of operations for this segment for fiscal 2003 as compared to fiscal 2002 and 2001. Management also believes that the inclusion of this information helps investors gain a better understanding of the Company’s core operating results and future prospects, consistent with how management measures and forecasts the Company’s performance, especially when comparing such results to previous periods or forecasts. The following pro forma financial tables were prepared based on available information using assumptions that the Company’s management believes are reasonable. The pro forma results do not purport to represent the actual results of operations that would have occurred if the Casual Male acquisition had occurred on February 4, 2001, and are not necessarily indicative of the results that may be achieved in the future.

22

Table of Contents

CASUAL MALE BUSINESS – SEGMENT OVERVIEW

Sales. Sales for fiscal 2003 for the Casual Male business, which includes sales from its e-commerce and catalog businesses, were $319.2 million as compared to sales of $242.7 million in fiscal 2002. This increase in sales of $76.5 million was due to the fact that fiscal 2003 represented a full year of revenue as compared to fiscal 2002, which reflected revenue only from May 14, 2002, the date of the Casual Male acquisition. Compared to pro forma sales for fiscal 2002 of $329.2 million, total sales for fiscal 2003 decreased 3.0% and comparable store sales decreased 1.3%. In addition, Casual Male discontinued its Repp catalog title during fiscal 2003, which resulted in an approximate 2% loss of sales as compared to fiscal 2002. On a pro-forma basis, fiscal 2002 comparable store sales decreased by 1% as compared to fiscal 2001. Comparable store sales include retail stores that have been open for at least one full fiscal year, together with e-commerce and catalog sales.

When the Company purchased the Casual Male business in May 2002, it believed that Casual Male had a strong and stable sales base, with significant potential to grow market share from its current 7% of the overall men’s big and tall apparel market. However, during the first quarter of fiscal 2003, the Company identified several merchandising issues, which the Company believes were negatively impacting sales. As discussed above under “Executive Summary,” the Company implemented several key merchandising and marketing initiatives geared towards improving sales trends and gaining market share. As a result of these initiatives, the Company was able to improve quarter over quarter comparable store sales in fiscal 2003.

Gross Margin. The Casual Male business gross margin rate for fiscal 2003 was 41.5% compared to 42.3% for fiscal 2002. This decrease of 0.8 percentage points was due to the deleveraging of occupancy costs of 1.0 percentage points offset slightly by improved merchandise margins of 0.2 percentage points. In spite of the decreases in sales that the Casual Male business experienced in the first half of fiscal 2003, the Company was able to maintain strong merchandise margins throughout fiscal 2003 as a result of stable initial margins and strong inventory management. Since fiscal 2001, Casual Male’s gross margin has dropped by approximately $10.0 million as a result of the decline in sales, which was primarily the result of decreases in comparable stores sales and the discontinuance of the Repp catalog. Casual Male’s recently developed and executed merchandising and marketing strategies were designed to broaden Casual Male’s appeal to its big and tall customer base, improve customer traffic into its stores, catalog and web site, and enhance gross margins.

The Company expects similar if not higher merchandise margin performance in fiscal 2004 as the Company continues to implement its merchandising initiatives, which appear to be having a positive impact on sales.

Selling, General and Administrative Expenses. Selling, general and administrative expenses (“SG&A”) as a percentage of sales were 34.2%, or $109.3 million, in fiscal 2003 and 32.6%, or $79.1 million, in fiscal 2002. On a pro forma basis, SG&A expenses as a percentage of sales in fiscal 2002 and fiscal 2001 were 35.2% and 37.1%, respectively. Since the completion of the Casual Male acquisition in May 2002, the Company has implemented several cost reduction and synergy initiatives, which have resulted in approximately $20.0 million in savings on an annualized basis. On a pro forma basis, since fiscal 2001, SG&A costs for the Casual Male business have declined by approximately 12%, on an average store basis.

The increase in SG&A costs as a percentage of sales of 1.6 percentage points for fiscal 2003 (34.2% of sales) from fiscal 2002 (32.6% of sales) is primarily due to the weighted impact of a full year’s of expenses which includes the first quarter,

23

Table of Contents

which historically runs at a higher percentage of sales then the remainder of the year. In addition, in the fourth quarter of fiscal 2003, the Company also started to increase its marketing spend in connection with its new merchandising initiatives and its George Foreman product lines.

For fiscal 2004, the Company plans to increase its marketing spend by approximately $7.0 million in anticipation of its launch of its George Foreman line of clothing. The majority of the incremental marketing expenditures is expected to be incurred in the first half of the year during its initial George Foreman marketing launch. Aside from this increase in marketing for fiscal 2004, the Company expects to see continued cost savings in other areas in fiscal 2004 as a result of its infrastructure initiatives, which were discussed more fully under “Business- Management Information Systems.”

Operating Income. The Casual Male business had operating income of $16.5 million in fiscal 2003 as compared to $17.6 million in fiscal 2002. On a pro forma basis, operating income for fiscal 2002 was $15.6 million, which includes an operating loss for the first quarter of fiscal 2002. The improvement in the Casual Male business’ operating income in fiscal 2003 as compared to fiscal 2002 and fiscal 2001, on a pro forma basis, was the result of significant cost reductions and strong merchandise margins. On an average store basis, the Casual Male business’ operating income has increased over 40% since fiscal 2001, in spite of a 7% decline in gross margins over the same period.

OTHER BRANDED APPAREL BUSINESS – SEGMENT OVERVIEW

Other Branded Apparel business includes the results of operations, on a continuing basis, for the Company’s Levi’^®/Dockers^® outlet stores and it Ecko Unltd.^® outlet and retail stores. In fiscal 2002, the Company announced that it would be winding down its Levi’s^®/Dockers^® business with the intention to eventually exit the business completely. During fiscal 2003, the Company closed an additional 25 Levi’s^®/Dockers^® Outlet stores. Accordingly, the operating results of these stores for fiscal 2003, 2002 and 2001 have been reclassified to discontinued operations and are discussed in more detail under “Consolidated Results of Operations- Discontinued Operations.”

At January 31, 2004, 58 Levi’s^®/Dockers^® outlet stores and 21 Ecko Unltd.^® stores were open and are included in the below table of operating results for the Other Branded Apparel business. The Company intends to close another approximate 25-30 Levi’s^®/Dockers^® stores during fiscal 2004, many of which will be closed during the fourth quarter.