Table of Contents

Index to Financial Statements

UNITED STATES

SECURITIES AND EXCHANGE COMMISSION

WASHINGTON, D.C. 20549

FORM 10-K

ANNUAL REPORT PURSUANT TO SECTION 13 OR 15(d)

OF THE SECURITIES EXCHANGE ACT OF 1934

CASUAL MALE RETAIL GROUP, INC.

(Exact name of registrant as specified in its charter)

(781) 828-9300

(Registrant’s telephone number, including area code)

Securities registered pursuant to Section 12(b) of the Act:

None

Securities registered pursuant to Section 12(g) of the Act:

Common Stock, $0.01 par value

(Title of each Class)

Indicate by check mark whether the registrant (1) has filed all reports required to be filed by Section 13 or 15(d) of the Securities Exchange Act of 1934 during the preceding 12 months (or for such shorter period that the registrant was required to file such reports), and (2) has been subject to such filing requirements for the past 90 days.  Yes  x  No  ¨

Indicate by check mark if disclosure of delinquent filers pursuant to Item 405 of Regulation S-K is not contained herein, and will not be contained, to the best of the registrant’s knowledge, in definitive proxy or information statements incorporated by reference in Part III of this Form 10-K or any amendment to this Form 10-K.  ¨

Indicate by check mark whether the registrant is an accelerated filer (as defined in Rule 12b-2 of the Act).  Yes  ¨  No  x

As of August 2, 2002, the aggregate market value of the common stock held by non-affiliates of the registrant was approximately $63.2 million, based on the last reported sale price on that date. Shares of common stock held by each executive officer and director and by each person who owns 10% or more of the outstanding common stock have been excluded on the basis that such persons may be deemed to be affiliates. This determination of affiliate status is not necessarily determinative for other purposes.

The registrant had 35,744,921 shares of Common Stock, $0.01 par value, outstanding as of April 25, 2003.

DOCUMENTS INCORPORATED BY REFERENCE

None.

Table of Contents

Index to Financial Statements

CASUAL MALE RETAIL GROUP, INC.

Index to Annual Report on Form 10-K

Year Ended February 1, 2003

2

Table of Contents

Index to Financial Statements

PART I.

Item 1.    Business

Casual Male Retail Group, Inc. (formerly known as Designs, Inc.) together with its subsidiaries (the “Company”) is the largest specialty apparel retailer of big and tall men’s apparel, with a presence throughout the United States and Puerto Rico. Unless the context indicates otherwise, all references to “we,” “our,” “ours,” “us” and “the Company” refer to Casual Male Retail Group, Inc. and its consolidated subsidiaries after giving effect to the May 2002 acquisition by Designs, Inc. of substantially all of the assets of Casual Male Corp. and certain of its subsidiaries and to Designs, Inc. and its consolidated subsidiaries prior to the completion of such acquisition.

HISTORY

The Company was incorporated in the State of Delaware in 1976 under the name Designs, Inc. Until fiscal 1996, the Company operated exclusively Levi Strauss & Co. branded apparel mall and outlet stores. In fiscal 1996, the Company began seeing limited growth opportunities with Levi Strauss & Co. and started to embark on several private label diversification strategies. These strategies ultimately were abandoned by the Company due to a variety of reasons including lack of brand recognition of its private labels by its customers. As a result of these failed strategies, the Company incurred approximately $86 million in operating losses during the fiscal years 1998, 1999 and 2000.

In October 1999, the stockholders of the Company elected a new board of directors, which subsequently appointed a new Chairman of the Company and management team. Under this new management, the Company saw significant cost reductions in both its store and overhead operations, resulting in a return to profitability in fiscal 2001. With this new effective low-cost structure in place, the Company renewed its strategy to become the premier operator in the outlet channel for other well-known branded manufacturers, in addition to Levi Strauss & Co.

In January 2002, the Company entered into a license agreement with Candie’s, Inc. (“Candie’s”), a leading designer and marketer of young women’s footwear, apparel and accessories. This license agreement enabled the Company to open and operate Candies^® branded retail stores in outlet malls and value centers throughout the United States. In the first quarter of fiscal 2003, the Company entered into a joint venture with EcKo Complex, LLC (“EcKo”), a leading design-driven lifestyle brand targeting young men and women with worldwide annual sales exceeding $200 million. Under this joint venture agreement, the Company will exclusively open and operate 75 EcKo Unltd.^® branded outlet stores throughout the United States over a six-year period.

In the first quarter of fiscal 2003, the Company continued to see erosion in its Levi’s^®/Dockers^® business. The Levi Strauss & Co. brands had been experiencing significant declines and loss of market share which were having a direct effect on the Company’s sales. In May 2002, with limited opportunity to expand its mature Levi’s^®/Dockers^® business, the Company completed its acquisition of the Casual Male business from Casual Male Corp., which, at the time of the acquisition, was the largest retailer of men’s clothing in the big and tall market in the United States. At a bankruptcy ordered auction, the Company acquired substantially all of the assets of Casual Male and certain of its subsidiaries for a purchase price of approximately $170 million, plus assumption of certain operating liabilities. Management of the Company saw substantial opportunities from this acquisition, summarized by the following key factors:

3

Table of Contents

Index to Financial Statements

In view of the significance of this acquisition to the growth and future identity of the Company, on August 8, 2002, the Company’s stockholders voted to change the Company’s name to “Casual Male Retail Group, Inc.”

Following the Casual Male acquisition, the Company re-evaluated its strategic initiatives. In light of the continued significant deterioration in its Levi’s^®/Dockers^® business and the less than expected success of its Candies^® outlet stores, the Company announced that it would downsize these businesses with the intention of eventually exiting them. This would enable management to focus its capital resources and energies on growing its more profitable Casual Male business. See “OTHER BRANDED APPAREL BUSINESSES-Levi’s^®/Dockers^® and Candies^® outlet businesses” for a description of the Company’s exit strategy.

Since the Casual Male acquisition, the Company has operated in two segments, its “Casual Male business” and its “Other Branded Apparel businesses”.

CASUAL MALE BUSINESS

The Casual Male business, which represents over 80% of the Company’s total annualized revenues, is a multi-channel retailer offering what it believes to be high quality casual wear to the big and tall customer. The Company offers its merchandise to customers through diverse selling and marketing channels, including over 465 retail and outlet stores, which operate under the names “Casual Male Big & Tall” and “Casual Male Big & Tall Outlet” stores, two catalogs and two e-commerce sites.

Market Position

As previously mentioned, the big and tall market is extremely fragmented and rapidly growing. The Company believes it has opportunities for additional growth from its current market share.

The Company believes the apparel demands of the big and tall customer historically have not been met through traditional men’s apparel stores. The big and tall customer frequently has difficulty finding an adequate selection of apparel in department and specialty stores. Furthermore, the big and tall market is made up of many small, local stores that typically have a narrow merchandise selection. As the largest retailer in this market, the Company has significant purchasing power and product selection through both size and style and is able to reach its customers in a variety of ways, including through its retail stores, catalogs and e-commerce sites.

Casual Male Customer

The Company markets and sells to big men with waist sizes from 40” to 70” and tall men who are 6’2” or taller. According to NPD, a national market research firm, approximately 64% of U.S. adults are overweight or obese, up from 56% just six years ago. The Company tries to appeal to all of its customers by providing multiple ways to shop. For customers that are somewhat uncomfortable in a traditional shopping environment, the Company offers them the opportunity to shop through catalogs as well as through the Company’s e-commerce sites. The Casual Male customer is often a destination shopper when it comes to purchasing apparel for himself.

As discussed above, the Company has also begun to market more to the “Under 30” big and tall customer, who has more discretionary income when it comes to purchasing clothing. By adding well-known branded youth apparel that is geared specifically to this age group, the Company hopes to attract this “Under 30” customer while still appealing to its core customer base.

4

Table of Contents

Index to Financial Statements

Multi-Channel Retailer

The Company provides convenient shopping alternatives to its customers through its stores, catalogs and e-commerce sites.

At February 1, 2003, the Company operated 409 Casual Male Big & Tall full-price retail stores, located primarily in strip centers, power centers or stand-alone locations. These stores target the middle-income customer seeking good quality at moderate prices. As a result these stores offer a strong selection of basic sportswear and other casual apparel. The average Casual Male store is approximately 3,400 square feet and has approximately $190 in sales per square foot.

At February 1, 2003, the Company operated 58 Casual Male Big & Tall outlet stores designed to offer a wide range of casual clothing for the big and tall customer at moderate prices (but generally lower prices than its retail stores) and lower prices. Most of the merchandise in the Company’s outlet stores is offered with the purchasing interests of the value-oriented customer in mind. These stores also serve as a consolidation point for clearance and other slow moving inventory. The average Casual Male outlet store is approximately 3,100 square feet and has approximately $185 in sales per square foot. Approximately 20% of the merchandise in the outlet stores originates from the retail stores.

The Company currently has two catalogs bearing the titles, “Casual Male Big & Tall” and “REPP Big & Tall,” respectively. The “Casual Male Big & Tall” catalog targets customers who have similar demographics to the Company’s Casual Male stores. The catalog carries an assortment of casual merchandise similar to what is available in the stores, but it also offers a broader selection of dress apparel, such as sportcoats, suit separates and other tailored clothing. The “REPP Big & Tall” catalog primarily carries proprietary, branded and designer apparel and is targeted to mid- to upper- income customers. The Company currently issues 17 editions of each catalog, which are released during selling seasons. In fiscal 2003, the Company circulated a total of 7.2 million catalogs.

Beginning in fiscal 2004, the Company plans to transition its REPP Big & Tall catalog customers to its Casual Male Big & Tall catalog. The REPP Big & Tall catalog was an extension of the REPP Ltd. Big & Tall stores, which were previously owned and subsequently closed by Casual Male Corp. The Company has seen a decrease in sales from this catalog due to the lack of name recognition and awareness of the REPP Big & Tall brand.

Prior to the Company’s acquisition of the Casual Male business, catalog mailings by Casual Male Corp. were not targeted to its retail customers. Since the acquisition, however, the Company’s strategy with regard to mailings has been to provide all of its customers with as many options to purchase its products as possible whether it be at a retail store, through a catalog or on the internet. Accordingly, the Company has increased its circulation database with the addition of over two million of its store customers and will be shipping catalogs to these customers beginning in fiscal 2004. As a result, total circulation is expected to increase more than 20% to nine million in fiscal 2004.

The Company also promotes catalog sales in its Casual Male Big & Tall stores. Expanded assortments of items, colors and sizes and brands in the catalogs complement and supplement the stores’ assortment. Such catalog orders are placed by store associates via an intra-net link and are shipped directly to customers from the Company’s distribution center in Canton, Massachusetts. This use of the catalog is intended to generate both additional sales and customer loyalty. In fiscal 2004, the Company plans to expand its catalog reach further by providing large supplies of catalogs to all stores and will test adding sample catalog merchandise to generate greater sales. In-bound calls and order fulfillment for the Company’s catalogs are currently handled at its distribution center. See “Distribution” below.

The Company currently has two e-commerce web sites: www.casualmale.com and www.reppbigandtall.com. These sites offer substantially the same merchandise as is offered in their respective catalogs. In addition to product offerings, www.casualmale.com contains a store locator directory and a corporate “investor relations” link. The Company uses the Ecometry and WebOrder software system that provides, among other things, electronic shopping carts, order taking, payment and security systems, order management, warehouse and shipping management and real-time inventory availability. The Company currently processes and fulfills orders from its web sites through its distribution center.

5

Table of Contents

Index to Financial Statements

Although to date its e-commerce web sites have generated relatively small sales volume, the Company sees the internet as a significant growth opportunity. Internet sales in fiscal 2003 increased 90% over the prior year, and the Company anticipates that internet sales will increase another 75% in fiscal 2004 to exceed $10 million. In addition, the Company has also signed an agreement with Amazon.com, the biggest internet seller of general merchandise, to launch Casual Male Big & Tall apparel shops on the Amazon.com web site by the fiscal 2004 back to school season. The Company expects the agreement with Amazon.com to provide further growth and brand identity for the Company’s Casual Male business.

Merchandise

The Company believes Casual Male offers a wide range of quality casual and dress apparel and footwear for big and tall men at moderate prices. Approximately 60% of Casual Male’s merchandise is basic or fashion-neutral items, such as jeans, casual slacks, tee-shirts and polo shirts. While the Company’s Casual Male business does carry several brand-name names, it also offers merchandise made specifically for Casual Male (“private label” merchandise), such as its Harbor Bay private label, which accounts for approximately 75% of Casual Male’s total merchandise.

In the second quarter of fiscal 2003, the Company entered into a sourcing agreement with the Kellwood Company. This agreement is intended to assist the Company in obtaining sufficient quantities and assortments of private label merchandise at a competitive cost. In fiscal 2003, merchandise purchased from the Kellwood Company was not material to total purchases for the year.

In order to attract the younger customer, who is between the ages of 17 and 30, the Company has started to introduce more youth-oriented branded merchandise into approximately 200 of its Casual Male Big & Tall retail stores. The Company believes that this big and tall age group has been under-serviced by apparel retailers and sees this target group as an opportunity to increase revenue growth. This merchandise includes such brands as EcKo Unltd.^®, Sean John, Rocawear, Pelle Pelle, Karl Kani, and Stacey Adams. Since the Company began to focus on this younger age group, youth wear sales have increased sizably.

The Company will continue to seek merchandise offerings which fulfill the needs of its customer base. During fiscal 2004, the Company expects to launch a custom fit program, by which customers can purchase at its retail stores, from its e-commerce sites or through its catalogs certain styles of pants and shirts custom made to each customer’s specific fit requirements. In addition, the Company intends to broaden its merchandise offerings in activewear by introducing certain branded merchandise in all channels and additional branded products via its e-commerce sites. Depending upon the needs and requirements of its customers, the Company expects to make other merchandise changes or additions in the future.

Marketing and Advertising

The Company’s marketing department creates and implements a wide variety of national, regional and local advertising, direct marketing and sales promotion programs. These television, radio and direct mail programs are designed to increase sales and customer awareness of the Casual Male brand name. Local store marketing activities occur on a regular basis and include store opening events and in-store promotion programs.

Advertising and marketing costs for the Casual Male businesses represent approximately 6% of the revenue from the Casual Male business. This includes advertising and all in-store signage for its more than 465 stores, creating and distributing nine million catalogs and supporting e-commerce efforts. Approximately 80% of the Company’s total marketing budget is spent on targeting the customer directly. The Company’s customer database contains more than 2.5 million names, together with their respective Casual Male purchase histories. In fiscal 2004, the Company expects to continue to test broad reach marketing programs in mass mediums in an effort to find cost effective methods for branding Casual Male Big & Tall and reaching its target customers.

Competitors of the Casual Male Business

The Casual Male business faces competition from a variety of sources, including department stores, specialty stores, discount and off-price retailers as well as other retailers that sell big and tall merchandise. While the Company has successfully competed on the basis of merchandise selection, favorable pricing, customer service and desirable store locations, there can be no assurances that other retailers will not adopt purchasing and marketing concepts similar to those

6

Table of Contents

Index to Financial Statements

of the Company. In addition, discount retailers with significant buying power, such as Wal-Mart and Target, represent an increasing source of competition for the Company.

Fiscal 2004 Casual Male Expansion Program

As previously mentioned, the Company expects to focus on seeking revenue growth in fiscal 2004 through increased circulation of its catalogs to its retail customers as well as continuing to grow its e-commerce operations, including its recently introduced apparel store on Amazon.com. The Company also plans to open seven new Casual Male Big & Tall retail stores, relocate six additional stores and open nine Casual Male Big & Tall outlets during the first half of fiscal 2004.

OTHER BRANDED APPAREL BUSINESSES

The Other Branded Apparel businesses consist of the Company’s Levi’s^®/Dockers^®, Candies^® and EcKo Unltd.^® outlet businesses.

Levi’s^®/Dockers^® and Candies^® outlet businesses

As discussed above, the Company announced in fiscal 2003 that it would be exiting its Levi’s^®/Dockers^® business and its Candies^® outlet business. The Company plans to close between 50 to 55 Levi’s^®/Dockers^® stores over the next 24 months, thereby reducing the sales base of its Levi’s^®/Dockers^® outlet business to less than 10% of the Company’s total sales. The Company expects that its remaining 30 Levi’s^®/Dockers^® stores will either be closed at the end of their respective lease terms, or otherwise be divested in a sales transaction. Until such closings or divestiture, these remaining stores will continue to offer an exclusive selection of Levi Strauss & Co. brands of merchandise, which include Levi’s^®, Dockers^® and Slates^® brands, at outlet prices. As such, the Company will continue to work with Levi Strauss & Co. on obtaining competitive costs for its products and, based on availability, will stock these remaining stores with a higher level of close-out merchandise, which will enable the Company to maintain its margin levels. The Company expects that these remaining stores will generate positive cash flow while they remain open.

The Company also decided to exit its Candies^® Outlet business by transferring the operations of seven of these stores to Candies, Inc. for the next 12 to 18 months. After that time, Candies, Inc. will either purchase the stores from the Company or the stores will be closed. The Company plans to convert four of the five stores remaining to either its Casual Male or EcKo Unltd.^® outlet store concepts during fiscal 2004.

The restructuring charges that the Company recorded in fiscal 2003 in connection with its decision to exit the Levi’s^®/Dockers^® business and the Candies^® Outlet business were approximately $41.3 million (which included an $8.0 million charge against certain deferred tax assets ) and are discussed in full under “Management’s Discussion and Analysis – Summary of Significant Fiscal 2003 Events.”

The EcKo Unltd.^® Business

Under its joint venture agreement with EcKo Complex, LLC, the Company, a 50.5% partner, owns and manages retail outlet stores bearing the name EcKo Unltd.^® and featuring EcKo^® branded merchandise. EcKo, a 49.5% partner, contributes to the joint venture the use of its trademark and the merchandise requirements, at cost, for the retail outlet stores. The Company contributes all real estate and operating requirements of the retail outlet stores, including but not limited to, the real estate leases, payroll needs and advertising. Each partner shares in the operating profits of the joint venture, after each partner has received reimbursement for its cost contributions. Under the terms of the agreement, the Company must maintain a prescribed store opening schedule and open 75 stores over a six-year period in order to maintain the joint venture’s exclusivity. At certain times during the term of the agreement, the Company may exercise a put option to sell its share of the retail joint venture to EcKo, and EcKo has an option to acquire the Company’s share of the retail joint venture at a price based on the performance of the retail outlet stores.

Sales performance of the six EcKo Unltd.^® outlet stores opened during fiscal 2003 exceeded $600 sales per square foot.

The Company believes that its EcKo^® Unltd. outlet stores represent an opportunity in the outlet marketplace for the underdeveloped young men’s and junior market because EcKo^® is believed to be a cross-over youth brand appealing to both urban and suburban youth with a core customer between the ages of 14 to 24. Accordingly, these stores provide a broad selection of merchandise ranging from hip-hop to extreme sports, and street-wear to fraternity wear, with a core menswear line consisting of fleece, twill and denim bottoms, wovens, printed tee shirts, knits and sweaters.

7

Table of Contents

Index to Financial Statements

Management believes that its EcKo Unltd.^® outlet stores compete with other outlet apparel retailers on the basis of selection of quality, service and price. The Company stresses product training with its sales staff and, with the assistance of merchandise materials from EcKo Complex, LLC, provides its sales personnel with substantial product knowledge training across all product lines.

Based on the performance through the end of fiscal 2003 of its six EcKo Unltd.^® outlet stores, the Company plans to open a total of 16 or 17 new EcKo Unltd.^® outlet stores during fiscal 2004, with 10 to 12 of such stores opening by the end of June 2003.

DISTRIBUTION

As part of the Company’s acquisition of the Casual Male business, the Company acquired the property in Canton, Massachusetts that served as Casual Male Corp.’s corporate offices and distribution center. At the end of fiscal 2003, the Company terminated the leases of its two other distribution centers, which were located in Orlando, Florida, and centralized all of its distribution operations in Canton.

The Company believes that having one centralized distribution facility minimizes the delivered cost of merchandise and maximizes the in-stock position of its stores. The Company believes that the centralized distribution system enables its stores to maximize selling space by reducing necessary levels of back-room stock carried in each store. In addition, the distribution center provides order fulfillment services for the Company’s e-commerce and catalog businesses.

In fiscal 2002, the Company partnered with United Parcel Services (“UPS”) to improve upon its distribution methods and reduce shipping costs as a result of not having to use third party trucking companies. By utilizing UPS, the Company is able to track all deliveries from the warehouse to the individual stores and gives the Company the added visibility to the status of in-transit shipments.

During fiscal 2003, the Company formulated different shipping methods to accommodate the distinct merchandise delivery needs of each of its retail stores. The Levi’s^®/Dockers^® and Ecko Unltd.^® outlet stores, which ship to stores multiple times per week, continue to partner with UPS thus taking advantage of UPS volume pricing incentives and the ability to track each shipment from the distribution center to the stores. The Casual Male stores utilize a network of third party LTL (less than full truck carriers), truckload and pool distribution carriers to maintain a once per week shipping schedule to its stores. This flexibility allows the Company to distribute merchandise on an as-needed basis to specific stores while maintaining lower freight costs. The Company continues to review its distribution methods to find more efficient and cost-effective ways to move merchandise to its stores.

In fiscal 2004, the Company expects to implement Manhattan Associates’ PKMS warehousing application for its distribution center systems which is expected to significantly streamline the Company’s distribution processes, enhance in-transit times, and significantly reduce distribution costs.

MANAGEMENT INFORMATION SYSTEMS

Since the Company’s acquisition of the Casual Male business, one of the Company’s primary goals has been to integrate the systems of the acquired business with those of the Company and to upgrade and enhance current systems where necessary. The Company has identified significant cost savings and synergies that can only be realized upon the completion of this integration and upgrade of the system infrastructure.

At the time of the acquisition, the Casual Male business operated primarily on an IBM mainframe platform. The mainframe based system included an internally supported enterprise retail system, and a human resource/payroll application. The Company’s then-existing e-commerce/catalog fulfillment infrastructure operated on a Hewlett-Packard environment on software from Ecometry, and the remainder of is business operated on an IBM 400 platform. The Company’s financial systems were supported by a client server environment.

Presently, the Company’s management information systems, which are located at the Company’s new corporate headquarters in Canton and at all of its retail stores, consist of a full range of retail merchandising and financial systems which include merchandise planning and reporting, distribution center processing, inventory allocation, sales reporting, and

8

Table of Contents

Index to Financial Statements

financial processing and reporting. Until integration is complete, the Company’s Casual Male business will continue to operate primarily on a mainframe platform, with the e-commerce/catalog business on the HP environment, and the remainder of the business on the IBM AS/400 platform

In February 2003, subsequent to the end of fiscal 2003, the Company completed the conversion of the Casual Male business to its Lawson Financial System. Management now has several tools from which to manage the business on a consolidated level in a more efficient and effective manner.

For fiscal 2004, the Company plans to implement the JDA E3 Advanced Replenishment system for the Casual Male business. This system will be integrated with the current Casual Male business mainframe platform in an effort to improve sales opportunities and better manage inventories of basic merchandise.

During fiscal 2004, the Company plans to complete the systems integration work to upgrade its merchandising systems to the JDA Portfolio Solutions, specifically the MMS Merchandise Management System, E3 Advanced Replenishment, Retail Ideas Data Warehouse, and Arthur Merchandise Planning and Advanced Allocation systems. In addition, the Company also plans to convert its distribution system to Manhattan Associates’ PKMS distribution system. Once the system is converted, the Company should be able to improve sales, better manage its inventory levels, and streamline operations.

Currently, all of the Company’s stores have point-of-sale terminals supplied by IBM, Fujitsu, and NCR. The Casual Male business is supported by a point-of-sale business application provided by Triversity, and the remainder of the Company’s business is supported by a point-of-sale business application by CRS. The point-of-sale applications capture daily transaction information by item, color and size. The Company utilizes barcode technology in tracking sales, inventory and pricing information. Communication between the corporate headquarters and all stores is facilitated on a daily basis through the use of an electronic mail system. The merchandising management systems are updated daily with all store transactions and provide daily sales, inventory, pricing and merchandise information and management reports to assist the Company in operating its retail business. The Company’s merchandising system applications also facilitate the placement and tracking of purchase orders and utilize electronic data interchange. The Company evaluates this information, together with weekly reports on merchandise statistics, prior to making decisions regarding reorders of fast-selling items and the allocation of merchandise.

The Company utilizes a Microsoft NT / Windows 2000 environment running on a local area network to communicate and work-share within its corporate headquarters. The Company also utilizes the services of ADP, an outside payroll processing provider, to prepare, distribute and report its weekly payroll. All of the Company’s payroll processing was consolidated onto the ADP processing effective January 1, 2003.

9

Table of Contents

Index to Financial Statements

SEASONALITY

Consistent with the retail industry, the Company’s business is seasonal. The Casual Male business traditionally generates the largest volume of its sales in June and the Christmas season. The Company’s outlet businesses, which include its Levi’s^®/Dockers^® and EcKo Unltd.^® outlet stores, traditionally generate more than 50% of its sales in the second half of the fiscal year.

TRADEMARKS/TRADEMARK LICENSE AGREEMENTS

The Company owns several servicemarks and trademarks relating to its Casual Male business, including Casual Male^® and Casual Male Big & Tall^®.

“Dockers^®,” “Levi’s^®” and “Slates^®” are registered trademarks of Levi Strauss & Co. “Candie’s^®” is a registered trademark of Candie’s, Inc. “EcKo^®” and “EcKo Unltd.^®” are registered trademarks of EcKo Complex, LLC.

The Company operates its Levi’s^®/Dockers^® business pursuant to a trademark license agreement with Levi Strauss & Co., which agreement was most recently amended in October 1998 (as amended, the “Levi Outlet License Agreement”). The Levi Outlet License Agreement authorizes the Company to use certain Levi Strauss & Co. trademarks in connection with the operation of the Company’s Levi’s^®/Dockers^® Outlet by Designs in 25 states in the eastern portion of the United States and in Puerto Rico.

The Company operates its EcKo Unltd.^® outlet stores pursuant to a joint venture agreement with EcKo Complex LLC which authorizes the Company to utilize the EcKo^® trademark in connection with the operations of its EcKo Unltd.^® outlet stores.

Employees

As of February 1, 2003, the Company employed approximately 4,340 associates, of whom 1,759 were full-time personnel. The Company hires additional temporary employees during the peak Fall and Holiday seasons. None of the Company’s employees is represented by any collective bargaining agreement.

Available Information

The Company’s corporate web site is www.cmrginc.com. The Company makes available free of charge through its web site its Annual Reports on Form 10-K, Quarterly Reports on Form 10-Q, Current Reports on Form 8-K and amendments to such reports filed or furnished pursuant to Section 13(a) of the Securities Exchange Act of 1934, as amended, as soon as reasonably practicable after the Company has electronically filed such material with, or furnished such materials to, the Securities and Exchange Commission.

10

Table of Contents

Index to Financial Statements

Item 2.     Properties

The Company’s corporate offices and distribution center are located at 555 Turnpike Street in Canton, Massachusetts. The facility is located on 37 acres of land and is owned by Designs Canton Property Corp., a wholly owned subsidiary of Designs Canton Holdings, Inc., which is an indirect wholly owed subsidiary of the Company. The property, which was acquired by the Company as part of its Casual Male acquisition, is subject to an outstanding mortgage of $11.6 million at February 1, 2003. The mortgage is secured by the real estate and the buildings. The property contains about 750,000 square feet, which includes approximately 150,000 square feet of office space.

The Company remains liable on its lease, which expires in January 2006, for its previous corporate offices located at 66 B Street, Needham, Massachusetts. With the exception of the Company’s loss-prevention subsidiary, LP Innovations, Inc., which occupies approximately 19,000 square feet of the Needham facility, the Company has relocated all operations to its Canton offices. Subsequent to year end, on March 1, 2003, the Company entered into a sub-lease agreement whereby it leases approximately 17,000 square feet of the total 80,000 square feet of leased space at the Needham facility. This sublease agreement expires in January 2006.

Until the end of fiscal 2003, the Company leased two distribution centers, totaling approximately 76,000 square feet, both of which were located in Orlando, Florida. As part of the Company’s integration of the Casual Male operations, the Company exercised its right to terminate both of these leases. The total cost to the Company to terminate these leases was approximately $76,000.

As of February 1, 2003, the Company operated 409 Casual Male Big and Tall retail stores, 58 Casual Male Big and Tall outlet stores, 82 Levi’s^®/Dockers^® Outlet stores and 6 EcKo Unltd.^® outlet stores. All of these stores are leased by the Company directly from shopping center owners. The store leases are generally five years in length and contain renewal options extending their terms to between 5 and 10 years.

The Company remains principally liable on seven store leases for its Candies^® outlet stores, the operations of which were transferred to Candies, Inc. at the end of fiscal 2003.

Sites for store expansion are selected on the basis of several factors intended to maximize the exposure of each store to the Company’s target customers. These factors include the demographic profile of the area in which the site is located, the types of stores and other retailers in the area, the location of the store within the center and the attractiveness of the store layout. The Company also utilizes financial models to project the profitability of each location using assumptions such as the center’s sales per square foot averages, estimated occupancy costs and return on investment requirements. The Company believes that its selection of locations enables the Company’s stores to attract customers from the general shopping traffic and to generate its own customers from surrounding areas.

See also “Management’s Discussion and Analysis of Financial Condition and Results of Operations – Liquidity and Capital Resources – Capital Expenditures.”

11

Table of Contents

Index to Financial Statements

Item 3.    Legal Proceedings

Although the Company is a party to litigation and claims arising in the course of its business, management does not expect the results of these actions to have a material adverse effect on the Company’s business or financial condition.

Item 4.    Submission of Matters to a Vote of Security Holders

No matters were submitted to a vote of the Company’s security holders during the fourth quarter of fiscal 2003.

12

Table of Contents

Index to Financial Statements

PART II.

Item 5.    Market for the Registrant’s Common Equity and Related Stockholder Matters

The Company’s Common Stock is listed for trading on the Nasdaq National Market under the symbol “CMRG.”

The following table sets forth, for the periods indicated, the high and low per share closing sales prices for the Common Stock, as reported on the Nasdaq consolidated reporting system.

As of April 25, 2003, based upon data provided by independent shareholder communication services and the transfer agent for the Common Stock, there were approximately 349 holders of record of Common Stock.

The Company has not paid and does not anticipate paying cash dividends on its common stock. For a description of financial covenants in the Company’s loan agreement that may restrict dividend payments, see Note C of the Notes to the Consolidated Financial Statements.

13

Table of Contents

Index to Financial Statements

Item 6.    Selected Financial Data

Selected Financial Data-Historical Results

14

Table of Contents

Index to Financial Statements

Selected 5 Year Financial Data of Casual Male

The following table provides historical financial information of the Casual Male business for the past five fiscal years, all of which except for fiscal 2003 is according to the reports of Casual Male Corp.

15

Table of Contents

Index to Financial Statements

Item 7.    Management’s Discussion and Analysis of Financial Condition and Results of Operations

SUMMARY OF SIGNIFICANT FISCAL 2003 EVENTS

Casual Male Acquisition

On May 14, 2002, the Company completed the acquisition of substantially all of the assets of Casual Male Corp. and certain of its subsidiaries (“Casual Male”) for a purchase price of approximately $170 million, plus the assumption of certain operating liabilities. The Company was selected as the highest and best bidder at a bankruptcy court ordered auction commencing on May 1, 2002 and concluding on May 2, 2002. The U.S. Bankruptcy Court for the Southern District of New York subsequently granted its approval of the acquisition on May 7, 2002.

Under the terms of the asset purchase agreement, the Company acquired substantially all of Casual Male’s assets, including, but not limited to, the inventory and fixed assets of approximately 475 retail store locations and various intellectual property. In addition, the Company assumed certain operating liabilities, including, but not limited to, existing retail store lease arrangements and the existing mortgage for Casual Male’s corporate office located in Canton, Massachusetts. See Note K of the Notes to the Consolidated Financial Statements for a further discussion of the transaction.

Corporate Name Change

In view of the significance of the Casual Male acquisition to the growth and future identity of the Company, at the Annual Meeting of Stockholders held on August 8, 2002, the Company’s stockholders approved the Board of Directors’ recommendation to change the Company’s name from “Designs, Inc.” to “Casual Male Retail Group, Inc.” The Company believes that the Casual Male business will be a primary future contributor to the Company’s overall business and that this name change was an important step to align the customer and investor identification of the Company with the Casual Male store concept. For the same reason, certain financial information for the Casual Male business is included in this Annual Report on Form 10-K.

Downsizing of the Company’s Levi’s^®/Dockers^® and Candies^® businesses

In fiscal 2003, based on continued decreases in sales and profitability from its Levi’s^®/Dockers^® business, the Company decided to downsize and ultimately exit its Levi’s^®/Dockers^® business completely. The Company plans to close between 50 to 55 stores over the next 24 months, thereby reducing the sales base of its Levi’s^®/Dockers^® outlet business to less than 10% of the Company’s total sales. After the Company completes the planned downsizing, the approximately 30 remaining stores will either be closed at the end of their respective lease terms or otherwise be divested in a sales transaction. In spite of continuing sales erosion in the Levi’s^®/Dockers^® business, the Company expects the remaining Levi’s^®/Dockers^® outlet stores to generate positive cash flow while the stores remain open.

In addition, in the fourth quarter of fiscal 2003, the Company also announced that it would exit the Candies^® Outlet business. The Company had entered into a trademark license agreement with Candies, Inc. at the end of fiscal 2002 and had planned to open up to 75 Candies^® Outlet stores. Since then, the Company has opened 12 of such stores; however, their performance through the end of fiscal 2003 had not achieved the Company’s expectations. Accordingly, the Company has transferred to Candies, Inc. the operations of seven of these 12 stores, whereby Candies, Inc. will operate such stores for the next 12 to 18 months, after which time, Candies, Inc. will either purchase the stores from the Company or the stores will be closed. The Company plans to convert four of the five remaining stores to either its Casual Male outlet or its EcKo Unltd.^® outlet store concepts during fiscal 2004.

As a result of the Company’s decision to exit its Levi’s^®/Dockers^® business and its Candies^® outlet business and to focus its attention and capital on the more profitable Casual Male business, the Company recorded a total of approximately $41.3 million in non-recurring charges, the details of which are discussed below under “Restructuring.”

16

Table of Contents

Index to Financial Statements

Proposed Transaction of Subsidiary

The business of one of the Company’s subsidiaries, LP Innovations, Inc. (“LPI”), a provider of loss prevention services and system solutions primarily to the retail industry, was acquired as part of the Casual Male acquisition. On November 27, 2002, the Company filed with the Securities and Exchange Commission a registration statement on Form S-1 to register common stock and warrants proposed to be distributed to the Company’s stockholders and optionholders, in anticipation of a spin-off of LPI to the Company’s stockholders, after which LPI will become a stand-alone and separately traded company. The Company expects to complete an offering to its stockholders during fiscal 2004.

SEGMENT REPORTING

With the Casual Male acquisition, the Company has redefined its business into two reportable business segments: (i) the Casual Male business and (ii) Other Branded Apparel businesses. See Note M of the Notes to Consolidated Financial Statements for a complete disclosure of the financial results of each segment. The Company’s “Other Branded Apparel businesses” segment, which is referred to in “Management’s Discussion and Analysis-Results of Operations,” includes the operations of the Company’s Levi’s^®/Dockers^® outlet stores, its Candies^® Outlet stores and its EcKo Unltd.^® outlet stores.

STORE CLOSINGS/DISCONTINUED OPERATIONS

The results of operations for 20 of the Company’s Levi’s^®/Dockers^® outlet stores which closed during fiscal 2003 and the results of operations of the Candies^® Outlet stores which are all part of the Company’s exit strategy, have been presented as discontinued operations in accordance with the provisions of Statement of Financial Accounting Standard (“SFAS”) No. 144, “Accounting for the Impairment or Disposal of Long-Lived Assets” (“SFAS No. 144”). The discounted operations of these stores include related write-downs for the impairment of assets and costs to liquidate inventory. Accordingly, the Company has restated all prior year results for these closed stores to conform with the current period presentation of discontinued operations.

RESULTS OF OPERATIONS

SALES—Continuing Operations

Sales for fiscal 2003, 2002 and 2001 were as follows, by segment:

Fiscal 2003 Compared to Fiscal 2002

Sales for fiscal 2003 were $398.3 million compared with sales of $174.1 million in fiscal 2002. Included in fiscal 2003 results were sales in the Casual Male business, since May 14, 2002, of $242.7 million. On a pro-forma basis, as if the Company had owned the Casual Male business for the full fiscal year, total sales would have decreased 1.0% to $329.2 million as compared to $332.4 million in the prior year. On the same pro forma basis, comparable store sales for the Casual Male business decreased 1%. Comparable store sales are those stores that have been open for at least one full fiscal year.

As previously discussed, the Company continued to experience significant sales declines in its Levi’s^®/Dockers^® business due to the erosion on the Levi Strauss & Co. brands and product availability issues, and, as a result, comparable store sales for the Levi’s^® business decreased 14% in fiscal 2003. Although the Company does not expect comparable store sales in its remaining Levi’s^®/Dockers^® outlet stores to improve significantly in fiscal 2004, the Company will continue to work with Levi Strauss & Co. on improving availability and selection of merchandise for these stores.

17

Table of Contents

Index to Financial Statements

Fiscal 2002 Compared to Fiscal 2001

Sales for fiscal 2002 were $174.1 million for the 52-week period compared with sales of $172.1 million for the 53-week period of fiscal 2001. Comparable store sales for fiscal 2002 decreased 3.9%. Fiscal 2002 was a difficult year for the retail industry due to the general economic conditions and the tragic events of September 11, 2001. In an effort to manage inventory levels and improve its sales trends, the Company significantly increased its levels of promotional activities during the second half of fiscal 2002. The impact of this aggressive promotional posture, although negative to the Company’s gross margin, significantly benefited its sales trends in the second half of fiscal 2002.

GROSS MARGIN—Continuing Operations

Below are the gross margin percentages by segment for fiscal 2003, 2002 and 2001. These percentages, which include occupancy costs, were calculated based on their respective segment sales:

Fiscal 2003 Compared to Fiscal 2002

Gross margin for fiscal 2003 of 31.4% increased 7.6 percentage points from 23.8% for fiscal 2002, solely due to the higher merchandise margins generated by the Company’s Casual Male business, which has been a significant benefit to the Company’s overall gross margin rates. The Casual Male business has a substantial portion of private label merchandise, which by its nature produces higher gross margins. In addition, limited competition in the big and tall market and the primarily basic nature of Casual Male’s sportswear merchandise assortments result in lower markdowns than the men’s apparel industry in general.

The Company’s increase in gross margin for fiscal 2003 was partially offset by approximately $11.1 million in inventory charges, which were recorded during the second and fourth quarters of fiscal 2003 in connection with the Company’s store closing plan. The charge of $11.1 million relates specifically to the liquidation of inventory for some of the Levi’s^®/Dockers^® outlet stores that the Company expects to close. The Company incurred costs of approximately $5.3 million related to the liquidation of inventory for 20 of its Levi’s^®/Dockers^® outlet stores which were closed during fiscal 2003 and for its exited Candies^® Outlet business. The $5.3 million is included in discontinued operations for fiscal 2003.

In addition, in an effort to improve sales and manage inventory levels in the Company’s Levi’s^®/Dockers^® business, gross margin for fiscal 2003 was also negatively impacted by the increased promotional activity incurred as compared to fiscal 2002.

Fiscal 2002 Compared to Fiscal 2001

Gross margin for fiscal 2002 of 23.8% decreased 4.3 percentage points from 28.1% for fiscal 2001, primarily due to the Company’s aggressive promotional activity, which resulted in a significantly higher markdown rate as compared to the prior year. The gross margin rate for fiscal 2002 was also negatively impacted by a decrease in initial margins related to increased costs of certain product lines. The Company was able to partially offset these decreases through its improvements in inventory shrink.

SELLING, GENERAL AND ADMINISTRATIVE EXPENSES—Continuing Operations

Selling, general and administrative expenses (“SG&A”) as a percentage of sales were 28.5% or $113.5 million in fiscal 2003, 20.7% or $36.0 million in fiscal 2002 and 22.1% or $38.1 million in fiscal 2001. The increase in SG&A as a percentage of sales for fiscal 2003 was due principally to the addition of the Casual Male cost structure to the Company’s existing low cost base. Since the completion of the Casual Male acquisition in May 2002, the Company has implemented several cost reduction and synergy initiatives, which have resulted in approximately $8 million of cost savings in fiscal 2003, or approximately $20 million on an annualized basis. The Company expects to ultimately realize, over the next 12 to

18

Table of Contents

Index to Financial Statements

18 months, an additional $5 million in cost savings on an annualized basis. Even prior to the Casual Male acquisition, the Company was actively implementing cost reduction initiatives which have resulted in the steady decrease in SG&A as a percentage of sales since fiscal 1999.

IMPAIRMENT OF ASSETS

The Company periodically assesses long-lived assets for impairment under SFAS No. 144, Accounting for the Impairment or Disposal of Long-Lived Assets. The Company reviews its long-lived assets for events or changes in circumstances that might indicate the carrying amount of the assets may not be recoverable. The Company assesses the recoverability of the assets by determining whether the carrying value of such assets over their respective remaining lives can be recovered through projected undiscounted future cash flows. The amount of impairment, if any, is measured based on projected discounted future cash flows using a discount rate reflecting the Company’s average cost of funds.

In fiscal 2003, the Company recorded an impairment charge of $14.4 million, which was included as part of the $41.3 million in restructuring charges recorded in the second and fourth quarters of fiscal 2003. See “Restructuring” below. Of the $14.4 million, approximately $2.5 million related to the 20 Levi’s^®/Dockers^® outlet stores closed in fiscal 2003 and the Candies^® outlet stores and accordingly is included in discontinued operations for fiscal 2003. The remaining $11.9 million relate to the balance of the Levi’s^®/Dockers^® outlet stores which will be closed in the future and is reflected in “Provision for impairment of assets, store closings, and severance” on the Consolidated Statement of Operations for fiscal 2003. No such charge was necessary for fiscal 2002. In fiscal 2001, the Company recorded an impairment charge of $837,000 for the write-down of fixed assets. The impairment charge related to stores whose expected cash flows from operations were not expected to exceed their net book value prior to the expiration of their expected lease term.

RESTRUCTURING

In the second and fourth quarters of fiscal 2003, the Company recorded a total of $41.3 million in restructuring charges, summarized in the table below:

The second quarter charge of approximately $11.1 million related to the Company’s decision to downsize its Levi’s^®/Dockers^® business and also included costs to integrate the newly acquired Casual Male business with the existing businesses of the Company. Of the $11.1 million, approximately $7.5 million related to the downsizing of the Levi’s^®/Dockers^® business and the remaining $3.6 million was integration costs associated with the combination of the Casual Male and Designs businesses. In the fourth quarter of fiscal 2003, the Company accelerated its plan to downsize the Levi’s^®/Dockers^® business and also to transfer its operations of the Candies^® Outlet business to Candies, Inc. Accordingly, the Company recorded an additional charge in the fourth quarter of approximately $30.2 million.

19

Table of Contents

Index to Financial Statements

Of the total restructuring charges of $41.3 million: i) $7.8 million relates to the 20 Levi’s^®/Dockers^® stores closed during fiscal 2003 and therefore, pursuant to SFAS No. 144, are included in loss from discontinued operations; ii) $25.5 million relates to the future closing of the balance of the Levi’s^®/Dockers^® stores and integration costs associated with combining the Casual Male business with the Company and, pursuant to SFAS No. 144, are included in loss from continuing operations; and iii) $8.0 million relates to the impairment of certain tax assets, which is shown under “Provision for income taxes” in the Consolidated Statement of Operations. The impairment of the Company’s tax assets does not affect its ability to utilize the $49.5 million federal net operating loss carryforwards to offset future taxable income, subject to certain annual limitations. Of the total restructuring charges of $41.3 million, only $5.5 million is expected to impact the Company’s cash liquidity, primarily in fiscal 2004.

DEPRECIATION AND AMORTIZATION—Continuing Operations

Depreciation and amortization expense for fiscal 2003 of $10.4 million increased as compared with $5.0 million in fiscal 2002, due to the addition of approximately $52.9 million in assets acquired in connection with the Casual Male acquisition, offset partially by the $14.4 million in impaired assets which the Company wrote off in connection with the restructuring and store closing charges recorded in fiscal 2003.

Depreciation and amortization expense for fiscal 2002 increased slightly to $5.0 million as compared to $4.9 million in fiscal 2001.

INTEREST EXPENSE, NET

Net interest expense for fiscal 2003 was $9.1 million compared with $1.9 million in fiscal 2002 and $1.8 million in fiscal 2001. The significant increase in interest expense in fiscal 2003 was due to the increased debt levels of the Company as a result of the Casual Male acquisition. Approximately $50 million in long term debt and additional borrowings of approximately $30.2 million were incurred in connection with the Casual Male acquisition. The Company also assumed a $12 million mortgage as part of the Casual Male acquisition. The average annual interest rate of this additional debt is approximately 8.3%. The increase in interest costs in fiscal 2002 as compared to fiscal 2001 was primarily a result of higher average borrowing levels offset partially by reduced interest rates under the Company’s credit facility. See “Liquidity and Capital Resources.”

INCOME TAX PROVISION

The income tax provision for fiscal 2003 includes a non-cash charge of $8.0 million attributable to an increase in the valuation allowance for the Company’s deferred tax assets related to the potential that certain federal and state tax assets may not be realized. The provision for fiscal 2002 also included an $8.0 million charge against the Company’s realizability of certain tax assets.

Pursuant to accounting rules, realization of the Company’s deferred tax assets, which relate principally to federal net operating loss carryforwards which expire from 2017 through 2023, is dependent on generating sufficient taxable income in the near term. The Company has not generated historical earnings in previous years and therefore has established a valuation allowance for its remaining deferred tax assets in the amount of $8.0 million. The impairment of the Company’s tax assets does not affect its ability to utilize its net operating loss carryforwards to offset any future taxable income, subject to certain annual limitations.

As of February 1, 2003, the Company has net operating loss carryforwards of $49.5 million for federal income tax purposes and $65.6 million for state income tax purposes that are available to offset future taxable income, subject to certain annual usage limitations, through fiscal year 2022. Additionally, the Company has alternative minimum tax credit carryforwards of $1.2 million, which are available to further reduce income taxes over an indefinite period. As a result of the Casual Male acquisition and the issuance of $82.5 million of additional equity, the utilization of approximately $33.7 million of the total $49.5 million of the net operating loss is limited to approximately $4.8 million each fiscal year.

In the third quarter of fiscal 2002, the Company and the Internal Revenue Service reached a final settlement on the audit of the Company’s federal income tax returns for fiscal years 1992 through 1996. In accordance with this settlement, the Company paid to the IRS a total of $1.5 million, including interest. The settlement of $1.5 million had no material impact on the Company’s results of operations for fiscal 2002 due to adequate provisions previously established by the Company.

20

Table of Contents

Index to Financial Statements

NET (LOSS) INCOME FROM CONTINUING OPERATIONS

For fiscal 2003, the Company had a net loss from continuing operations of $30.7 million as compared to a net loss of $9.5 million in fiscal 2002 and net income of $2.0 million in fiscal 2001. The primary reason for the decrease in net loss from continuing operations during fiscal 2003 was due to $25.5 million of the $41.3 million in restructuring charges and the deferred tax asset impairment of $8.0 million that were charged to continuing operations in fiscal 2003. After excluding the effect of these charges, the Company’s net income from continuing operations was $2.8 million.

DISCONTINUED OPERATIONS

Discontinued operations for 2003 resulted in a loss of $8.1 million compared to net income of $1.6 million in fiscal 2002 and net income of $1.2 million (net of $0.8 million for income taxes) in fiscal 2001. Due to the consolidated loss for the Company in fiscal 2003 and 2002, no tax benefit or provision was realized on discontinued operations for either year. Included in discontinued operations are the results of operations for fiscal 2003, 2002 and 2001 of the Company’s Candies^® outlet stores and its 20 Levi’s^®/Dockers^® outlet stores closed during fiscal 2003 pursuant to the Company’s exit strategy. There were no results of operations for the Candies^® Outlet business in fiscal 2002 and fiscal 2001.

Below is a summary of the results of operations for these discontinued operations for the past three fiscal years:

21

Table of Contents

Index to Financial Statements

NET INCOME (LOSS)

The Company reported a net loss of $38.8 million, or $(1.54) per basic and diluted share, in fiscal 2003 compared to a net loss of $7.9 million, or $(0.54) per basic and diluted share, in fiscal 2002 and net income of $3.2 million, or $0.20 per basic and diluted share, in fiscal 2001. Included in fiscal 2003 were restructuring charges, including the write-down of tax assets of $8.0 million, of $41.3 million related primarily to the Company’s decision to downsize and exit its Levi’s^®/Dockers^® business and its Candies^® business. See “Restructuring” above for further discussion.

SEASONALITY

A comparison of sales in each quarter of the past three fiscal years is presented below. The amounts shown are not necessarily indicative of actual trends, since such amounts also reflect the addition of new stores and the remodeling and closing of others during these periods. Historically, the Company has experienced seasonal fluctuations in revenues and income, exclusive of non-recurring charges, with increases occurring during the Company’s third and fourth quarters as a result of “Fall” and “Holiday” seasons. A comparison of quarterly sales, gross profit, net income (loss) per share for the past two fiscal years is presented in Note N of the Notes to Consolidated Financial Statements.

Below are the quarterly sales, on a pro forma basis for the past two years of the Casual Male business, assuming that it was purchased on the first day of fiscal 2002.

LIQUIDITY AND CAPITAL RESOURCES

The Company’s primary cash needs are for working capital (essentially inventory requirements) and capital expenditures. Specifically, the Company’s capital expenditure program includes projects for new store openings, downsizing or

22

Table of Contents

Index to Financial Statements

combining existing stores, and improvements and integration of its systems infrastructure. The Company expects that cash flow from operations, external borrowings and trade credit will enable it to finance its current working capital and expansion requirements.

The following table sets forth financial data regarding the Company’s liquidity position at the end of the past three fiscal years:

The Company has financed its working capital requirements, store expansion program, stock repurchase programs and acquisitions with cash flow from operations, external borrowings, and proceeds from equity offerings. Cash provided by operations was $20.6 million, $0.6 million and $6.3 million in fiscal 2003, 2002 and 2001, respectively.

Cash flow provided by operations in fiscal 2003 increased approximately $20.0 million from fiscal 2002, primarily due to the Casual Male business which, since its acquisition in May 2002, has contributed approximately $17.6 million in operating income. This positive cash flow in fiscal 2003 was primarily used to finance the Company’s capital expenditures and reduce its borrowings under the Credit Facility (as defined below).

During fiscal 2003, after excluding the effects of the restructuring charges, the Company’s operating income from continuing operations was approximately $12.0 million. In addition, the Company significantly reduced its operating expense levels by approximately $20 million on an annualized basis, $8.0 million of which was realized in fiscal 2003. The Company expects that its operating income levels in fiscal year 2004 will therefore improve, providing sufficient cash flow to fund working capital requirements and capital expenditure levels.

In addition to cash flow from operations, the Company’s other primary source of working capital is its credit facility with Fleet Retail Finance, Inc. (the “Credit Facility”). The Credit Facility, which was most recently amended on May 14, 2002 in connection with the financing of the Casual Male acquisition, provides for a total commitment of $120.0 million with a $20.0 million carve-out for standby and documentary letters of credit and continues to be secured by a lien on all of the Company’s assets. The Credit Facility, which expires May 14, 2005, will continue to provide the Company with its on-going working cash needs. The Company’s ability to borrow under the Credit Facility is determined using an availability formula based on eligible assets. The Company borrowed approximately $30.2 million under the Credit Facility in May 2002 to partially fund its Casual Male acquisition. The average outstanding borrowings under the Credit Facility for fiscal 2003 was $55.5 million and the highest borrowing level during the year was $85.0 million. At February 1, 2003, the Company had outstanding borrowings of approximately $55.6 million and unused availability of approximately $17.6 million. The Company anticipates that cash flow from operations and availability under the Credit Facility will be sufficient to meet all debt service requirements.

In addition to approximately $30.2 million of financing from the Credit Facility, the Company financed the Casual Male acquisition through the issuances of $82.5 million of additional equity and approximately $50 million of new long-term debt with detachable warrants. The Company also assumed a mortgage note in the principal amount of approximately $12.2 million for Casual Male Corp.’s corporate headquarters and distribution center in Canton, Massachusetts. See Note K to the Notes to Consolidated Financial Statements for a complete discussion of the funding.

As previously discussed, the Company is in the process of exiting its Levi’s^®/Dockers^® outlet stores. To the extent the Company is not able to terminate its leases with various landlords, the Company remains liable to meet its obligations under the existing lease agreements. See Note E to Consolidated Financial Statement.

Cash flow provided by operations in fiscal 2002 decreased approximately $5.7 million from fiscal 2001, primarily due to lower earnings. The Company used the cash proceeds from operations of $0.6 million and borrowings under the Credit

23

Table of Contents

Index to Financial Statements

Facility to finance its store openings and remodeling program and other capital requirements of approximately $4.0 million, net of landlord allowances received. Correspondingly, the Company’s net borrowing position increased by approximately $3.4 million to $27.8 million at February 2, 2002 as compared to the prior year.

INVENTORY

At February 1, 2003, total inventories of the Company were $103.2 million as compared to $57.7 million at February 2, 2002. Of the $103.2 million, approximately $62.3 million was related to the Casual Male business and $40.9 million was related to the Other Branded Apparel businesses. Total inventory at February 1, 2003 is net of approximately $11.1 million in inventory reserves related to the exiting of its Levi’s^®/Dockers^® outlet stores.

In the first quarter of fiscal 2002, the Company changed its method of determining the cost of inventories from the last-in, first-out (“LIFO”) method to the first-in, first-out (“FIFO”) method. Management believes that the FIFO method better measures the current value of inventories and provides a more appropriate matching of revenues and expenses. In the current low-inflationary environment, management believes that the use of the FIFO method more accurately reflects the Company’s financial position. The effect of the change from LIFO to FIFO was immaterial to the financial results of the prior reporting periods of the Company and therefore did not require retroactive restatement of results for such prior periods.

CAPITAL EXPENDITURES

During fiscal 2003, the Company incurred approximately $14.5 million in capital expenditures, of which approximately $8.4 million was used towards store capital, $5.4 million was related to systems integration and infrastructure upgrades, and the remaining $0.7 million was miscellaneous capital at the corporate level.

During fiscal 2003, the Company opened six Casual Male Big & Tall retail stores and one Casual Male Big Tall outlet store. The Company also opened six EcKo Unltd.^® outlet stores at an average build-out cost of approximately $175,000 per store. The Company’s plan for fiscal 2004 is to open seven new Casual Male Big & Tall retail stores, relocate six additional stores and, during the first half of fiscal 2004, open nine Casual Male Big & Tall outlets. The Company also plans to open a total of 16 new EcKo Unltd.^® outlet stores, 10 to 12 of which are scheduled to open by the end of June 2003.

Capital expenditures for fiscal 2004 are expected to be between $12.0 million and $15.0 million, of which $6.0 million relates to the store expansion plans discussed above. The remaining planned capital for fiscal 2004 is primarily related to new merchandising and distribution systems, and maintenance level requirements.

CRITICAL ACCOUNTING POLICIES

The Company’s financial statements are based on the application of significant accounting policies, many of which require management to make significant estimates and assumptions (see Note A to the Notes to Consolidated Financial Statements). The Company believes that the following items involve some of the more critical judgments in the application of its accounting policies that currently affect its financial condition and results of operations.

Inventory.    The Company records inventory at the lower of cost or market on a FIFO basis. The Company reserves for obsolescence based on the difference between the weighted average cost of the inventory and the estimated market value of the inventory based on assumptions of future demand and market conditions. If actual market conditions are less favorable than those projected by management, additional reserves may be required.

Impairment of Long-Lived Assets.    The Company reviews its long-lived assets for impairment when indicators of impairment are present and the undiscounted cash flow estimated to be generated by those assets is less than the assets’ carrying amount. The Company evaluates its long-lived assets for impairment at a store level for all its retail locations. If actual market conditions are less favorable than management’s projections, future write-offs may be necessary.

Goodwill and Intangibles:    In connection with the Casual Male acquisition, the Company recorded goodwill and intangibles of approximately $81.4 million. In accordance with Statement of Financial Accounting Standards (“SFAS”) No. 142, Goodwill and Other Intangible Assets (“SFAS No. 142”), the Company will, at least annually, evaluate these assets for impairment by analyzing the estimated fair value based on present value of discounted cash flows and will write off the amount of any goodwill or intangible in excess of its fair value.

24

Table of Contents

Index to Financial Statements

Deferred Taxes.    The Company records a valuation allowance to reduce its deferred tax assets to the amount that is more likely than not to be realized. The Company has considered estimated future taxable income and ongoing tax planning strategies in assessing the amount needed for the valuation allowance. At February 1, 2003, the Company’s deferred tax assets are fully reserved.

RECENT ACCOUNTING PRONOUNCEMENTS

In June 2001, the Financial Accounting Standards Board (‘FASB”) issued SFAS No. 143, Accounting for Asset Retirement Obligations, which requires that the fair value of a liability for an asset retirement obligation be recognized in the period in which it is incurred if a reasonable estimate of fair value can be made. The associated asset retirement costs would be capitalized as part of the carrying amount of the long-lived asset and depreciated over the life of the asset. The liability is accreted at the end of each period through charges to operating expense. If the obligation is settled for other than the carrying amount of the liability, a company will recognize a gain or loss on settlement. The provisions of SFAS No. 143 are effective for fiscal years beginning after June 15, 2002. The Company does not expect implementation of SFAS No. 143 to have a significant effect on our results of operation or consolidated financial condition.

In April 2002, the FASB issued SFAS No. 145, Rescission of FASB Statements No. 4, 44, and 64, Amendment of FASB Statement No. 13, and Technical Corrections as of April 2002, which is effective for fiscal years beginning after May 15, 2002. SFAS No. 145 rescinds SFAS No. 4 which required that all gains and losses from extinguishment of debt be aggregated, and if material, classified as an extraordinary item. As a result, gains and losses from debt extinguishment are to be classified as extraordinary only if they meet the criteria set forth in Accounting Principles Board Opinion No. 30, Reporting the Results of Operations—Reporting the Effects of Disposal of a Segment of a Business, and Extraordinary, Unusual and Infrequently Occurring Events and Transactions. SFAS No. 145 also requires that sale-leaseback accounting be used for capital lease modifications with economic effects similar to sale-leaseback transactions. The Company does not expect implementation of SFAS No. 145 to have a significant effect on our results of operation or consolidated financial condition.

In November 2002, the FASB published interpretation No. 45, Guarantor’s Accounting and Disclosure Requirements for Guarantees, Including Indirect Guarantees of Indebtedness of Others (“FIN 45”). FIN 45 expands on the accounting guidance of SFAS Nos. 5, 57 and 107 and incorporates without change the provisions of FASB interpretation No. 34, which FIN 45 superseded. FIN 45 elaborates on the existing disclosure requirements for most guarantees, including loan guarantees such as standby letters of credit. FIN 45 also provides that at the time a company issues a guarantee, such company must recognize an initial liability for the fair value, or market value, of the obligations it assumes under such guarantee and must disclose that information in its interim and annual financial statements. The initial recognition and initial measurement provisions apply on a prospective basis to guarantees issued or modified after December 31, 2002, regardless of the guarantor’s fiscal year-end. The disclosure requirements in FIN 45 are effective for financial statements with respect to interim or annual periods ending after December 15, 2002. The adoption of the provisions of FIN 45 in fiscal 2003 did not have an impact on the Company’s results of operations or disclosures.

In December 2002, the FASB issued SFAS No. 148, Accounting for Stock-Based Compensation—Transition and Disclosure, an amendment of SFAS No. 123. The purpose of the amendment is to enable companies that choose to adopt the fair-value-based method of accounting for stock-based compensation to report the full effect of employee stock options in their financial statements immediately upon adoption. The Company will continue to apply the disclosure-only provisions of SFAS No. 123. The transition provisions are effective for fiscal years ending after December 15, 2002. The Company adopted the annual disclosure provision of SFAS No. 148 in fiscal 2003. The interim disclosure provisions are effective for financial reports containing condensed financial statements for interim periods beginning after December 15, 2002, and the Company will adopt such provisions in the first quarter of fiscal 2004.

In January 2003, the FASB’s Emerging Issues Task Force, or EITF, reached a consensus on Issue 02-16, Accounting by a Customer (including a Reseller) for Cash Consideration Received from a Vendor. Issue 02-16 provides guidance on how a customer should account for cash consideration received from a vendor. The transition provisions apply prospectively to arrangements entered into or modified subsequent to December 31, 2002 and would require all amounts received from vendors to be accounted for as a reduction of the cost of the products purchased unless certain criteria are met to allow presentation as a reduction of advertising expense. The Company will adopt the provisions of Issue 02-16 in the first

25

Table of Contents

Index to Financial Statements

quarter of fiscal 2004. The Company currently accounts for vendor consideration, which is not material to the Company, as a component of cost of goods sold and therefore believes that the adoption of Issue 02-16 will not have a material impact on its results of operations.

In January 2003, the FASB published interpretation No. 46, Consolidation of Variable Interest Entities (“FIN 46”), to clarify the conditions under which assets, liabilities and activities of another entity should be consolidated into the financial statements of a company. FIN 46 requires the consolidation of a variable interest entity (including a special purpose entity such as that utilized in an accounts receivable securitization transaction) by a company that bears the majority of the risk of loss from the variable interest entity’s activities or is entitled to receive a majority of the variable interest entity’s residual returns or both. The provisions of FIN 46 are required to be adopted in fiscal 2004. The Company does not believe the adoption of FIN 46 will have a material impact on its financial position or results of operations.

EFFECTS OF INFLATION

Although the Company’s operations are influenced by general economic trends, the Company does not believe that inflation has had a material effect on the results of its operations in the last three fiscal years.

RISKS AND UNCERTAINTIES

Certain statements contained in this Annual Report on Form 10-K constitute “forward-looking statements” within the meaning of the United States Private Securities Litigation Reform Act of 1995. In some cases, forward-looking statements can be identified by the use of forward-looking terminology such as “may,” “will,” “estimate,” “intend,” “plan,” “continue,” “believe,” “expect” or “anticipate” or the negatives thereof, variations thereon or similar terminology. The forward-looking statements contained in this Annual Report are generally located in the material set forth under the headings “Management’s Discussion and Analysis of Financial Condition and Results of Operations” and “Business,” but may be found in other locations as well. These forward-looking statements generally relate to plans and objectives for future operations and are based upon management’s reasonable estimates of future results or trends. The forward-looking statements in this Annual Report should not be regarded as a representation by the Company or any other person that the objectives or plans of the Company will be achieved. Numerous factors could cause the Company’s actual results to differ materially from such forward-looking statements. The Company encourages readers to refer to the Company’s Current Report on Form 8-K, previously filed with the Securities and Exchange Commission on September 17, 2002, which identifies certain risks and uncertainties that may have an impact on future earnings and the direction of the Company.

All subsequent written and oral forward-looking statements attributable to the Company or to persons acting on the Company’s behalf are expressly qualified in their entirety by the foregoing. These forward-looking statements speak only as of the date of the document in which they are made. The Company disclaims any obligation or undertaking to provide any updates or revisions to any forward-looking statement to reflect any change in its expectations or any change in events, conditions or circumstances in which the forward-looking statement is based.

26

Table of Contents

Index to Financial Statements

Item 7A.    Quantitative and Qualitative Disclosures About Market Risk

In the normal course of business, the financial position and results of operations of the Company are routinely subject to a variety of risks, including market risk associated with interest rate movements on borrowings. The Company regularly assesses these risks and has established policies and business practices to protect against the adverse effects of these and other potential exposures. The Company utilizes cash from operations and its Credit Facility to fund its working capital needs. The Company’s Credit Facility is not used for trading or speculative purposes. In addition, the Company has available letters of credit as sources of financing for its working capital requirements. Borrowings under the Credit Facility, which expires in May 2005, bear interest at variable rates based on FleetBoston, N.A.’s prime rate or the London Interbank Offering Rate (“LIBOR”). These interest rates at February 1, 2003 were 4.75% for prime based borrowings and included various LIBOR contracts with interest rates ranging from 4.3% to 4.6%. Based upon sensitivity analysis as of February 1, 2003, a 50 basis point increase in interest rates would result in a potential increase in interest expense of approximately $278,000.

27

Table of Contents

Index to Financial Statements

Item 8.    Financial Statements and Supplementary Data

CASUAL MALE RETAIL GROUP, INC.

INDEX TO CONSOLIDATED FINANCIAL STATEMENTS

28

Table of Contents

Index to Financial Statements

MANAGEMENT’S RESPONSIBILITY FOR FINANCIAL REPORTING

The integrity and objectivity of the financial statements and the related financial information in this report are the responsibility of the management of the Company. The financial statements have been prepared in conformity with generally accepted accounting principles and include, where necessary, the best estimates and judgments of management.

The Company maintains a system of internal accounting control designed to provide reasonable assurance, at appropriate cost, that assets are safeguarded, transactions are executed in accordance with management’s authorization and the accounting records provide a reliable basis for the preparation of the financial statements. The system of internal accounting control is regularly reviewed by management and improved and modified as necessary in response to changing business conditions.

The Audit Committee of the Board of Directors, consisting solely of outside directors, meets periodically with management and the Company’s independent auditors to review matters relating to the Company’s financial reporting, the adequacy of internal accounting control and the scope and results of audit work. The independent auditors have free access to the Audit Committee.

Ernst & Young LLP, independent auditors, have been engaged to examine the financial statements of the Company for the fiscal year ended February 1, 2003. The Report of Ernst & Young, Independent Auditors expresses an opinion as to the fair presentation of the financial statements in accordance with generally accepted accounting principles and is based on an audit conducted in accordance with auditing standards generally accepted in the United States.