Help with determining how to classify my data

I have a set of data I would like to o something with. The what is still unknown at this time, which is part of my problem. I am hoping the folks who are much more learned than I at this point can provide some direction.

I have a set of data representing looks and products.

A look has an id, description, date, image, and a set of products. A product has an id, image, description and a set of tags (shirt, pants, women, etc). (The JSON representation of an actual data point is below).

I would like to be able to determine which products are the most similar to each other, using either the image itself, the descriptions, or the tags (or some combo of all). Additionally, it would be cool to know which looks are similar based on the products they have.

I think some sort of classifier would work for the descriptions, I can do something with the tags, but overall I am a bit lost trying to figure out the best way to tackle this problem.

I would greatly appreciate some suggestion, feedback, direction, etc. Not asking for specific implementations, but if anyone has them, I am more than happy to listen! I can also make this dataset available to others who might be interested, and I can provide it in different formats.

Thanks!
Birch

{
  "lookId": 42944443,
  "lookImageUrl": "https://i.shopstyle-cdn.com/i/951e7433-2fb5-4115-81a1-2017112b506f/1c8-226/image.jpeg",
  "description": "Last day in NY!  #wearitloveit #getthelook #todaysdetails #currentlywearing #lookoftheday #summerstyle #mylook #ootd",
  "internalTags": [
    "statementbag",
    "rawhemdenim",
    "blackskinnyjeans",
    "rippedjeans",
    "sweater",
    "whitesweater",
    "handbags-category-look"
  ],
  "publishDate": 1505401593164,
  "products": [
    {
      "productId": 670010674,
      "productImageUrl": "https://img.shopstyle-cdn.com/pim/20/c0/20c0bde1a768ddaed3033108b4615800.jpg",
      "description": "Polished studs and color-blocked styling give trend-savvy appeal to a leather crossbody bag in a structured, saddle-inspired silhouette. Style Name:Kate Spade New York Madison Stewart Street - Byrdie Studded Leather Satchel. Style Number: 5388751. Available in stores.",
      "tags": [
        "Women",
        "Satchels",
        "Handbags"
      ]
    },
    {
      "productId": 657125727,
      "productImageUrl": "https://img.shopstyle-cdn.com/pim/50/29/5029f82237adb62f34042e178eb2ac9c.jpg",
      "description": "061 The ‘90s are alive and well in this high/low chenille sweater from Silence + Noise. Made from a so-soft chenille knit in a wide-cut relaxed-fit silhouette. Topped with a wide v-neckline that can be worn on or off the shoulder. Finished with a high/low hem. <ul> <li>Polyester</li> <li>Hand wash</li> <li>Imported</li> <li>Model in Grey is 5'9\" and wearing size Small</li> <li>Measurements taken from size Small</li> <li>Chest: 51\"</li> <li>Length: 16\"</li> </ul>",
      "tags": [
        "Women",
        "Sweaters"
      ]
    },
    {
      "productId": 664490701,
      "productImageUrl": "https://img.shopstyle-cdn.com/pim/6e/01/6e017625ac9b61c47be3c2088750cc81.jpg",
      "description": "Eyelash lace trims the raw cuffs of these MOTHER jeans. Shredded holes at the knees. 5-pocket styling. Button closure and zip fly. <ul> <li>Fabric: Stretch denim.</li> <li>84% cotton/10% modal/4% polyester/2% elastane.</li> <li>Wash cold or dry clean.</li> <li>Made in the USA.</li> <li>Imported materials.</li> <li>Measurements</li> <li>Rise: 10.75in / 27cm</li> <li>Inseam: 26in / 66cm</li> <li>Leg opening: 9.5in / 24cm</li> <li>Measurements from size 27</li> </ul>",
      "tags": [
        "Women",
        "Stretch",
        "Jeans",
        "Skinny"
      ]
    },
    {
      "productId": 669117224,
      "productImageUrl": "https://img.shopstyle-cdn.com/pim/c9/a4/c9a40e446d3603ed0a5b5c9184cba632.jpg",
      "description": "Frayed fringe trims the topline of a satin pump in a pointy-toe silhouette that's glamorous and sophisticated. Brand: SAM EDELMAN. Style Name:Sam Edelman Halan Pointy Toe Pump (Women). Style Number: 5409472. Available in stores.",
      "tags": [
        "Women",
        "Pumps",
        "Shoes"
      ]
    },
    {
      "productId": 671813451,
      "productImageUrl": "https://img.shopstyle-cdn.com/pim/a6/25/a625ecc8508edb6461609128f6cb4385.jpg",
      "description": "Sweater by ASOS Collection, Fine knit, Crew neck, Ribbed cuffs and hem, Dropped shoulders, Cropped cut, Regular fit - true to size, Machine wash, 33% Acrylic, 27% Polyester, 23% Nylon, 17% Viscose, Our model wears a UK 8/EU 36/US 4 and is 170cm/5'7 tall. Score a wardrobe win no matter the dress code with our ASOS Collection own-label collection. From polished prom to the after party, our London-based design team scour the globe to nail your new-season fashion goals with need-right-now dresses, outerwear, shoes and denim in the coolest shapes and fits.",
      "tags": [
        "Women",
        "Sweaters"
      ]
    }
  ]
}

My goal with this FYI is to build a recommendation system that works similar to the following.

Take a picture of a ‘look’, like below:


and doing the following.

Detect the garments and accessories (i.e. shirt, shoes, glasses, bag, etc.) then match those images with images in a catalog to recommend which products to buy for that look.