{
  "id": 144585,
  "title": "Class Weights ... Pay Attention to Under-Represented Classes",
  "url": "/competitions/flower-classification-with-tpus/discussion/144585",
  "author_name": "",
  "post_date": "2020-04-19T17:50:26.749799300Z",
  "votes": 2,
  "comment_count": 2,
  "views": 0,
  "content": "<p>```\nimport math</p>\n\n<p>data = load_dataset(TRAINING_FILENAMES,labeled=True, ordered=True).batch(1024).cache().prefetch(AUTO)\ny_train = next(iter(data.unbatch().map(lambda image, label: label).batch(NUM_TRAINING_IMAGES))).numpy()\n```</p>\n\n<p><code>\ntrain_agg = np.asarray([[label, (y_train == index).sum()] for index, label in enumerate(CLASSES)])\ndf = pd.DataFrame(train_agg).rename(columns={0:\"Flower\",1:\"Count\"})\ndf.Count = df.Count.astype(int)\ndf = df.sort_values(by=\"Count\",ascending=False)\ndf = df.reset_index()\nlabels_dict = dict(zip(df.Flower,df.Count))\n</code></p>\n\n<p>```</p>\n\n<h1>mu in \"create_class_weight\" is a dampening parameter that could be tuned</h1>\n\n<p>def create_class_weight(labels_dict, mu=0.5):\n    total = np.sum(list(labels_dict.values()))\n    keys = labels_dict.keys()\n    class_weight = dict()\n    class_weight_log = dict()</p>\n\n<pre><code>for key in keys:\n    score = total / float(labels_dict[key])\n    score_log = math.log(mu * total / float(labels_dict[key]))\n    class_weight[key] = round(score, 2) if score &gt; 1.0 else round(1.0, 2)\n    class_weight_log[key] = round(score_log, 2) if score_log &gt; 1.0 else round(1.0, 2)\n\nreturn class_weight, class_weight_log\n</code></pre>\n\n<h1>Class abundance for protein dataset</h1>\n\n<p>labels_dict = new_dict</p>\n\n<p>print('\\nTrue class weights:')\nprint(create_class_weight(labels_dict)[0])\nprint('\\nLog-dampened class weights:')\nprint(create_class_weight(labels_dict)[1])\n```</p>\n\n<p>Pass this class weights values to <strong>model.fit()</strong> function for the <strong>class_weight</strong> parameter.</p>",
  "messages": [
    {
      "id": "813493",
      "postDate": "04/19/2020 17:50:26",
      "content": "<p>```\nimport math</p>\n\n<p>data = load_dataset(TRAINING_FILENAMES,labeled=True, ordered=True).batch(1024).cache().prefetch(AUTO)\ny_train = next(iter(data.unbatch().map(lambda image, label: label).batch(NUM_TRAINING_IMAGES))).numpy()\n```</p>\n\n<p><code>\ntrain_agg = np.asarray([[label, (y_train == index).sum()] for index, label in enumerate(CLASSES)])\ndf = pd.DataFrame(train_agg).rename(columns={0:\"Flower\",1:\"Count\"})\ndf.Count = df.Count.astype(int)\ndf = df.sort_values(by=\"Count\",ascending=False)\ndf = df.reset_index()\nlabels_dict = dict(zip(df.Flower,df.Count))\n</code></p>\n\n<p>```</p>\n\n<h1>mu in \"create_class_weight\" is a dampening parameter that could be tuned</h1>\n\n<p>def create_class_weight(labels_dict, mu=0.5):\n    total = np.sum(list(labels_dict.values()))\n    keys = labels_dict.keys()\n    class_weight = dict()\n    class_weight_log = dict()</p>\n\n<pre><code>for key in keys:\n    score = total / float(labels_dict[key])\n    score_log = math.log(mu * total / float(labels_dict[key]))\n    class_weight[key] = round(score, 2) if score &gt; 1.0 else round(1.0, 2)\n    class_weight_log[key] = round(score_log, 2) if score_log &gt; 1.0 else round(1.0, 2)\n\nreturn class_weight, class_weight_log\n</code></pre>\n\n<h1>Class abundance for protein dataset</h1>\n\n<p>labels_dict = new_dict</p>\n\n<p>print('\\nTrue class weights:')\nprint(create_class_weight(labels_dict)[0])\nprint('\\nLog-dampened class weights:')\nprint(create_class_weight(labels_dict)[1])\n```</p>\n\n<p>Pass this class weights values to <strong>model.fit()</strong> function for the <strong>class_weight</strong> parameter.</p>",
      "rawMarkdown": "```\nimport math\n\ndata = load_dataset(TRAINING_FILENAMES,labeled=True, ordered=True).batch(1024).cache().prefetch(AUTO)\ny_train = next(iter(data.unbatch().map(lambda image, label: label).batch(NUM_TRAINING_IMAGES))).numpy()\n```\n\n```\ntrain_agg = np.asarray([[label, (y_train == index).sum()] for index, label in enumerate(CLASSES)])\ndf = pd.DataFrame(train_agg).rename(columns={0:\"Flower\",1:\"Count\"})\ndf.Count = df.Count.astype(int)\ndf = df.sort_values(by=\"Count\",ascending=False)\ndf = df.reset_index()\nlabels_dict = dict(zip(df.Flower,df.Count))\n```\n\n```\n#mu in \"create_class_weight\" is a dampening parameter that could be tuned\ndef create_class_weight(labels_dict, mu=0.5):\n    total = np.sum(list(labels_dict.values()))\n    keys = labels_dict.keys()\n    class_weight = dict()\n    class_weight_log = dict()\n\n    for key in keys:\n        score = total / float(labels_dict[key])\n        score_log = math.log(mu * total / float(labels_dict[key]))\n        class_weight[key] = round(score, 2) if score &gt; 1.0 else round(1.0, 2)\n        class_weight_log[key] = round(score_log, 2) if score_log &gt; 1.0 else round(1.0, 2)\n\n    return class_weight, class_weight_log\n\n# Class abundance for protein dataset\nlabels_dict = new_dict\n\nprint('\\nTrue class weights:')\nprint(create_class_weight(labels_dict)[0])\nprint('\\nLog-dampened class weights:')\nprint(create_class_weight(labels_dict)[1])\n```\n\nPass this class weights values to **model.fit()** function for the **class_weight** parameter.",
      "votes": null
    },
    {
      "id": "813497",
      "postDate": "04/19/2020 17:53:31",
      "content": "<p><strong>Output for TRAINING_FILES+VAlIDATION_FILES combined,</strong></p>\n\n<p><strong>True class weights:</strong></p>\n\n<p><code>{'iris': 16.3, 'wild rose': 17.17, 'wild geranium': 18.13, 'common dandelion': 22.65, 'rose': 27.72, 'sunflower': 27.72, 'daisy': 30.21, 'common tulip': 32.67, 'morning glory': 41.68, 'pink primrose': 46.91, 'spear thistle': 48.43, 'buttercup': 48.86, 'windflower': 49.0, 'yellow iris': 56.19, 'petunia': 63.57, 'wallflower': 74.17, 'water lily': 76.23, 'frangipani': 83.16, 'cyclamen ': 87.12, 'foxglove': 91.47, 'lotus': 91.47, 'azalea': 93.02, 'snapdragon': 93.55, 'hibiscus': 95.17, 'bougainvillea': 97.43, 'camellia': 100.4, 'marigold': 101.64, 'thorn apple': 102.27, 'magnolia': 106.23, 'passion flower': 106.92, 'artichoke': 107.61, 'anthurium': 108.32, 'primula': 111.25, 'columbine': 113.55, 'bee balm': 115.14, 'poinsettia': 115.95, 'sweet william': 116.77, 'hippeastrum ': 120.18, 'bird of paradise': 121.07, 'carnation': 121.07, 'wild pansy': 121.96, 'gazania': 122.87, 'clematis': 126.65, 'mallow': 127.64, 'watercress': 132.78, 'pincushion flower': 132.78, 'daffodil': 132.78, 'tree poppy': 136.07, 'black-eyed susan': 137.21, 'king protea': 138.36, 'balloon flower': 141.94, 'gaura': 143.17, 'monkshood': 147.01, 'tiger lily': 147.01, 'toad lily': 148.33, 'grape hyacinth': 151.06, 'globe thistle': 152.45, 'corn poppy': 153.88, 'lenten rose': 175.16, 'barberton daisy': 200.79, 'geranium': 203.27, 'sword lily': 203.27, 'pink-yellow dahlia': 219.53, 'bishop of llandaff': 219.53, 'cape flower': 222.5, 'californian poppy': 231.9, 'purple coneflower': 231.9, 'peruvian lily': 257.27, 'fritillary': 265.56, 'canna lily': 279.07, \"colt's foot\": 294.02, 'mexican petunia': 310.66, 'lilac hibiscus': 343.02, 'ruby-lipped cattleya': 350.32, 'orange dahlia': 357.93, 'bromelia': 374.2, 'stemless gentian': 374.2, 'osteospermum': 392.02, 'trumpet creeper': 411.62, 'desert-rose': 411.62, 'tree mallow': 433.29, 'japanese anemone': 445.0, 'silverbush': 457.36, 'cautleya spicata': 470.43, 'giant white arum lily': 484.26, 'great masterwort': 484.26, 'hard-leaved pocket orchid': 498.94, 'blackberry lily': 514.53, 'pink quill': 531.13, 'garden phlox': 531.13, 'blanket flower': 531.13, 'love in the mist': 548.83, 'spring crocus': 609.81, 'prince of wales feathers': 609.81, 'sweet pea': 609.81, 'globe-flower': 609.81, 'canterbury bells': 633.27, 'cosmos': 633.27, 'fire lily': 658.6, 'red ginger': 658.6, 'siam tulip': 686.04, 'bolero deep blue': 715.87, 'moon orchid': 715.87, 'alpine sea holly': 715.87}</code></p>\n\n<p><strong>Log-dampened class weights:</strong></p>\n\n<p><code>{'iris': 2.1, 'wild rose': 2.15, 'wild geranium': 2.2, 'common dandelion': 2.43, 'rose': 2.63, 'sunflower': 2.63, 'daisy': 2.72, 'common tulip': 2.79, 'morning glory': 3.04, 'pink primrose': 3.16, 'spear thistle': 3.19, 'buttercup': 3.2, 'windflower': 3.2, 'yellow iris': 3.34, 'petunia': 3.46, 'wallflower': 3.61, 'water lily': 3.64, 'frangipani': 3.73, 'cyclamen ': 3.77, 'foxglove': 3.82, 'lotus': 3.82, 'azalea': 3.84, 'snapdragon': 3.85, 'hibiscus': 3.86, 'bougainvillea': 3.89, 'camellia': 3.92, 'marigold': 3.93, 'thorn apple': 3.93, 'magnolia': 3.97, 'passion flower': 3.98, 'artichoke': 3.99, 'anthurium': 3.99, 'primula': 4.02, 'columbine': 4.04, 'bee balm': 4.05, 'poinsettia': 4.06, 'sweet william': 4.07, 'hippeastrum ': 4.1, 'bird of paradise': 4.1, 'carnation': 4.1, 'wild pansy': 4.11, 'gazania': 4.12, 'clematis': 4.15, 'mallow': 4.16, 'watercress': 4.2, 'pincushion flower': 4.2, 'daffodil': 4.2, 'tree poppy': 4.22, 'black-eyed susan': 4.23, 'king protea': 4.24, 'balloon flower': 4.26, 'gaura': 4.27, 'monkshood': 4.3, 'tiger lily': 4.3, 'toad lily': 4.31, 'grape hyacinth': 4.32, 'globe thistle': 4.33, 'corn poppy': 4.34, 'lenten rose': 4.47, 'barberton daisy': 4.61, 'geranium': 4.62, 'sword lily': 4.62, 'pink-yellow dahlia': 4.7, 'bishop of llandaff': 4.7, 'cape flower': 4.71, 'californian poppy': 4.75, 'purple coneflower': 4.75, 'peruvian lily': 4.86, 'fritillary': 4.89, 'canna lily': 4.94, \"colt's foot\": 4.99, 'mexican petunia': 5.05, 'lilac hibiscus': 5.14, 'ruby-lipped cattleya': 5.17, 'orange dahlia': 5.19, 'bromelia': 5.23, 'stemless gentian': 5.23, 'osteospermum': 5.28, 'trumpet creeper': 5.33, 'desert-rose': 5.33, 'tree mallow': 5.38, 'japanese anemone': 5.4, 'silverbush': 5.43, 'cautleya spicata': 5.46, 'giant white arum lily': 5.49, 'great masterwort': 5.49, 'hard-leaved pocket orchid': 5.52, 'blackberry lily': 5.55, 'pink quill': 5.58, 'garden phlox': 5.58, 'blanket flower': 5.58, 'love in the mist': 5.61, 'spring crocus': 5.72, 'prince of wales feathers': 5.72, 'sweet pea': 5.72, 'globe-flower': 5.72, 'canterbury bells': 5.76, 'cosmos': 5.76, 'fire lily': 5.8, 'red ginger': 5.8, 'siam tulip': 5.84, 'bolero deep blue': 5.88, 'moon orchid': 5.88, 'alpine sea holly': 5.88}</code></p>",
      "rawMarkdown": "**Output for TRAINING_FILES+VAlIDATION_FILES combined,**\n\n**True class weights:**\n\n`{'iris': 16.3, 'wild rose': 17.17, 'wild geranium': 18.13, 'common dandelion': 22.65, 'rose': 27.72, 'sunflower': 27.72, 'daisy': 30.21, 'common tulip': 32.67, 'morning glory': 41.68, 'pink primrose': 46.91, 'spear thistle': 48.43, 'buttercup': 48.86, 'windflower': 49.0, 'yellow iris': 56.19, 'petunia': 63.57, 'wallflower': 74.17, 'water lily': 76.23, 'frangipani': 83.16, 'cyclamen ': 87.12, 'foxglove': 91.47, 'lotus': 91.47, 'azalea': 93.02, 'snapdragon': 93.55, 'hibiscus': 95.17, 'bougainvillea': 97.43, 'camellia': 100.4, 'marigold': 101.64, 'thorn apple': 102.27, 'magnolia': 106.23, 'passion flower': 106.92, 'artichoke': 107.61, 'anthurium': 108.32, 'primula': 111.25, 'columbine': 113.55, 'bee balm': 115.14, 'poinsettia': 115.95, 'sweet william': 116.77, 'hippeastrum ': 120.18, 'bird of paradise': 121.07, 'carnation': 121.07, 'wild pansy': 121.96, 'gazania': 122.87, 'clematis': 126.65, 'mallow': 127.64, 'watercress': 132.78, 'pincushion flower': 132.78, 'daffodil': 132.78, 'tree poppy': 136.07, 'black-eyed susan': 137.21, 'king protea': 138.36, 'balloon flower': 141.94, 'gaura': 143.17, 'monkshood': 147.01, 'tiger lily': 147.01, 'toad lily': 148.33, 'grape hyacinth': 151.06, 'globe thistle': 152.45, 'corn poppy': 153.88, 'lenten rose': 175.16, 'barberton daisy': 200.79, 'geranium': 203.27, 'sword lily': 203.27, 'pink-yellow dahlia': 219.53, 'bishop of llandaff': 219.53, 'cape flower': 222.5, 'californian poppy': 231.9, 'purple coneflower': 231.9, 'peruvian lily': 257.27, 'fritillary': 265.56, 'canna lily': 279.07, \"colt's foot\": 294.02, 'mexican petunia': 310.66, 'lilac hibiscus': 343.02, 'ruby-lipped cattleya': 350.32, 'orange dahlia': 357.93, 'bromelia': 374.2, 'stemless gentian': 374.2, 'osteospermum': 392.02, 'trumpet creeper': 411.62, 'desert-rose': 411.62, 'tree mallow': 433.29, 'japanese anemone': 445.0, 'silverbush': 457.36, 'cautleya spicata': 470.43, 'giant white arum lily': 484.26, 'great masterwort': 484.26, 'hard-leaved pocket orchid': 498.94, 'blackberry lily': 514.53, 'pink quill': 531.13, 'garden phlox': 531.13, 'blanket flower': 531.13, 'love in the mist': 548.83, 'spring crocus': 609.81, 'prince of wales feathers': 609.81, 'sweet pea': 609.81, 'globe-flower': 609.81, 'canterbury bells': 633.27, 'cosmos': 633.27, 'fire lily': 658.6, 'red ginger': 658.6, 'siam tulip': 686.04, 'bolero deep blue': 715.87, 'moon orchid': 715.87, 'alpine sea holly': 715.87}`\n\n**Log-dampened class weights:**\n\n`{'iris': 2.1, 'wild rose': 2.15, 'wild geranium': 2.2, 'common dandelion': 2.43, 'rose': 2.63, 'sunflower': 2.63, 'daisy': 2.72, 'common tulip': 2.79, 'morning glory': 3.04, 'pink primrose': 3.16, 'spear thistle': 3.19, 'buttercup': 3.2, 'windflower': 3.2, 'yellow iris': 3.34, 'petunia': 3.46, 'wallflower': 3.61, 'water lily': 3.64, 'frangipani': 3.73, 'cyclamen ': 3.77, 'foxglove': 3.82, 'lotus': 3.82, 'azalea': 3.84, 'snapdragon': 3.85, 'hibiscus': 3.86, 'bougainvillea': 3.89, 'camellia': 3.92, 'marigold': 3.93, 'thorn apple': 3.93, 'magnolia': 3.97, 'passion flower': 3.98, 'artichoke': 3.99, 'anthurium': 3.99, 'primula': 4.02, 'columbine': 4.04, 'bee balm': 4.05, 'poinsettia': 4.06, 'sweet william': 4.07, 'hippeastrum ': 4.1, 'bird of paradise': 4.1, 'carnation': 4.1, 'wild pansy': 4.11, 'gazania': 4.12, 'clematis': 4.15, 'mallow': 4.16, 'watercress': 4.2, 'pincushion flower': 4.2, 'daffodil': 4.2, 'tree poppy': 4.22, 'black-eyed susan': 4.23, 'king protea': 4.24, 'balloon flower': 4.26, 'gaura': 4.27, 'monkshood': 4.3, 'tiger lily': 4.3, 'toad lily': 4.31, 'grape hyacinth': 4.32, 'globe thistle': 4.33, 'corn poppy': 4.34, 'lenten rose': 4.47, 'barberton daisy': 4.61, 'geranium': 4.62, 'sword lily': 4.62, 'pink-yellow dahlia': 4.7, 'bishop of llandaff': 4.7, 'cape flower': 4.71, 'californian poppy': 4.75, 'purple coneflower': 4.75, 'peruvian lily': 4.86, 'fritillary': 4.89, 'canna lily': 4.94, \"colt's foot\": 4.99, 'mexican petunia': 5.05, 'lilac hibiscus': 5.14, 'ruby-lipped cattleya': 5.17, 'orange dahlia': 5.19, 'bromelia': 5.23, 'stemless gentian': 5.23, 'osteospermum': 5.28, 'trumpet creeper': 5.33, 'desert-rose': 5.33, 'tree mallow': 5.38, 'japanese anemone': 5.4, 'silverbush': 5.43, 'cautleya spicata': 5.46, 'giant white arum lily': 5.49, 'great masterwort': 5.49, 'hard-leaved pocket orchid': 5.52, 'blackberry lily': 5.55, 'pink quill': 5.58, 'garden phlox': 5.58, 'blanket flower': 5.58, 'love in the mist': 5.61, 'spring crocus': 5.72, 'prince of wales feathers': 5.72, 'sweet pea': 5.72, 'globe-flower': 5.72, 'canterbury bells': 5.76, 'cosmos': 5.76, 'fire lily': 5.8, 'red ginger': 5.8, 'siam tulip': 5.84, 'bolero deep blue': 5.88, 'moon orchid': 5.88, 'alpine sea holly': 5.88}`",
      "votes": null
    },
    {
      "id": "818485",
      "postDate": "04/23/2020 23:27:18",
      "content": "<p>Did you test using it and what did you notice ? Improvement in score ? More training time ?</p>",
      "rawMarkdown": "Did you test using it and what did you notice ? Improvement in score ? More training time ?",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 813497,
      "author_name": "rhtsingh",
      "author_url": "",
      "post_date": "04/19/2020 17:53:31",
      "content": "<p><strong>Output for TRAINING_FILES+VAlIDATION_FILES combined,</strong></p>\n\n<p><strong>True class weights:</strong></p>\n\n<p><code>{'iris': 16.3, 'wild rose': 17.17, 'wild geranium': 18.13, 'common dandelion': 22.65, 'rose': 27.72, 'sunflower': 27.72, 'daisy': 30.21, 'common tulip': 32.67, 'morning glory': 41.68, 'pink primrose': 46.91, 'spear thistle': 48.43, 'buttercup': 48.86, 'windflower': 49.0, 'yellow iris': 56.19, 'petunia': 63.57, 'wallflower': 74.17, 'water lily': 76.23, 'frangipani': 83.16, 'cyclamen ': 87.12, 'foxglove': 91.47, 'lotus': 91.47, 'azalea': 93.02, 'snapdragon': 93.55, 'hibiscus': 95.17, 'bougainvillea': 97.43, 'camellia': 100.4, 'marigold': 101.64, 'thorn apple': 102.27, 'magnolia': 106.23, 'passion flower': 106.92, 'artichoke': 107.61, 'anthurium': 108.32, 'primula': 111.25, 'columbine': 113.55, 'bee balm': 115.14, 'poinsettia': 115.95, 'sweet william': 116.77, 'hippeastrum ': 120.18, 'bird of paradise': 121.07, 'carnation': 121.07, 'wild pansy': 121.96, 'gazania': 122.87, 'clematis': 126.65, 'mallow': 127.64, 'watercress': 132.78, 'pincushion flower': 132.78, 'daffodil': 132.78, 'tree poppy': 136.07, 'black-eyed susan': 137.21, 'king protea': 138.36, 'balloon flower': 141.94, 'gaura': 143.17, 'monkshood': 147.01, 'tiger lily': 147.01, 'toad lily': 148.33, 'grape hyacinth': 151.06, 'globe thistle': 152.45, 'corn poppy': 153.88, 'lenten rose': 175.16, 'barberton daisy': 200.79, 'geranium': 203.27, 'sword lily': 203.27, 'pink-yellow dahlia': 219.53, 'bishop of llandaff': 219.53, 'cape flower': 222.5, 'californian poppy': 231.9, 'purple coneflower': 231.9, 'peruvian lily': 257.27, 'fritillary': 265.56, 'canna lily': 279.07, \"colt's foot\": 294.02, 'mexican petunia': 310.66, 'lilac hibiscus': 343.02, 'ruby-lipped cattleya': 350.32, 'orange dahlia': 357.93, 'bromelia': 374.2, 'stemless gentian': 374.2, 'osteospermum': 392.02, 'trumpet creeper': 411.62, 'desert-rose': 411.62, 'tree mallow': 433.29, 'japanese anemone': 445.0, 'silverbush': 457.36, 'cautleya spicata': 470.43, 'giant white arum lily': 484.26, 'great masterwort': 484.26, 'hard-leaved pocket orchid': 498.94, 'blackberry lily': 514.53, 'pink quill': 531.13, 'garden phlox': 531.13, 'blanket flower': 531.13, 'love in the mist': 548.83, 'spring crocus': 609.81, 'prince of wales feathers': 609.81, 'sweet pea': 609.81, 'globe-flower': 609.81, 'canterbury bells': 633.27, 'cosmos': 633.27, 'fire lily': 658.6, 'red ginger': 658.6, 'siam tulip': 686.04, 'bolero deep blue': 715.87, 'moon orchid': 715.87, 'alpine sea holly': 715.87}</code></p>\n\n<p><strong>Log-dampened class weights:</strong></p>\n\n<p><code>{'iris': 2.1, 'wild rose': 2.15, 'wild geranium': 2.2, 'common dandelion': 2.43, 'rose': 2.63, 'sunflower': 2.63, 'daisy': 2.72, 'common tulip': 2.79, 'morning glory': 3.04, 'pink primrose': 3.16, 'spear thistle': 3.19, 'buttercup': 3.2, 'windflower': 3.2, 'yellow iris': 3.34, 'petunia': 3.46, 'wallflower': 3.61, 'water lily': 3.64, 'frangipani': 3.73, 'cyclamen ': 3.77, 'foxglove': 3.82, 'lotus': 3.82, 'azalea': 3.84, 'snapdragon': 3.85, 'hibiscus': 3.86, 'bougainvillea': 3.89, 'camellia': 3.92, 'marigold': 3.93, 'thorn apple': 3.93, 'magnolia': 3.97, 'passion flower': 3.98, 'artichoke': 3.99, 'anthurium': 3.99, 'primula': 4.02, 'columbine': 4.04, 'bee balm': 4.05, 'poinsettia': 4.06, 'sweet william': 4.07, 'hippeastrum ': 4.1, 'bird of paradise': 4.1, 'carnation': 4.1, 'wild pansy': 4.11, 'gazania': 4.12, 'clematis': 4.15, 'mallow': 4.16, 'watercress': 4.2, 'pincushion flower': 4.2, 'daffodil': 4.2, 'tree poppy': 4.22, 'black-eyed susan': 4.23, 'king protea': 4.24, 'balloon flower': 4.26, 'gaura': 4.27, 'monkshood': 4.3, 'tiger lily': 4.3, 'toad lily': 4.31, 'grape hyacinth': 4.32, 'globe thistle': 4.33, 'corn poppy': 4.34, 'lenten rose': 4.47, 'barberton daisy': 4.61, 'geranium': 4.62, 'sword lily': 4.62, 'pink-yellow dahlia': 4.7, 'bishop of llandaff': 4.7, 'cape flower': 4.71, 'californian poppy': 4.75, 'purple coneflower': 4.75, 'peruvian lily': 4.86, 'fritillary': 4.89, 'canna lily': 4.94, \"colt's foot\": 4.99, 'mexican petunia': 5.05, 'lilac hibiscus': 5.14, 'ruby-lipped cattleya': 5.17, 'orange dahlia': 5.19, 'bromelia': 5.23, 'stemless gentian': 5.23, 'osteospermum': 5.28, 'trumpet creeper': 5.33, 'desert-rose': 5.33, 'tree mallow': 5.38, 'japanese anemone': 5.4, 'silverbush': 5.43, 'cautleya spicata': 5.46, 'giant white arum lily': 5.49, 'great masterwort': 5.49, 'hard-leaved pocket orchid': 5.52, 'blackberry lily': 5.55, 'pink quill': 5.58, 'garden phlox': 5.58, 'blanket flower': 5.58, 'love in the mist': 5.61, 'spring crocus': 5.72, 'prince of wales feathers': 5.72, 'sweet pea': 5.72, 'globe-flower': 5.72, 'canterbury bells': 5.76, 'cosmos': 5.76, 'fire lily': 5.8, 'red ginger': 5.8, 'siam tulip': 5.84, 'bolero deep blue': 5.88, 'moon orchid': 5.88, 'alpine sea holly': 5.88}</code></p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 818485,
      "author_name": "ibrahimsherify",
      "author_url": "",
      "post_date": "04/23/2020 23:27:18",
      "content": "<p>Did you test using it and what did you notice ? Improvement in score ? More training time ?</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "813493": "```\nimport math\n\ndata = load_dataset(TRAINING_FILENAMES,labeled=True, ordered=True).batch(1024).cache().prefetch(AUTO)\ny_train = next(iter(data.unbatch().map(lambda image, label: label).batch(NUM_TRAINING_IMAGES))).numpy()\n```\n\n```\ntrain_agg = np.asarray([[label, (y_train == index).sum()] for index, label in enumerate(CLASSES)])\ndf = pd.DataFrame(train_agg).rename(columns={0:\"Flower\",1:\"Count\"})\ndf.Count = df.Count.astype(int)\ndf = df.sort_values(by=\"Count\",ascending=False)\ndf = df.reset_index()\nlabels_dict = dict(zip(df.Flower,df.Count))\n```\n\n```\n#mu in \"create_class_weight\" is a dampening parameter that could be tuned\ndef create_class_weight(labels_dict, mu=0.5):\n    total = np.sum(list(labels_dict.values()))\n    keys = labels_dict.keys()\n    class_weight = dict()\n    class_weight_log = dict()\n\n    for key in keys:\n        score = total / float(labels_dict[key])\n        score_log = math.log(mu * total / float(labels_dict[key]))\n        class_weight[key] = round(score, 2) if score &gt; 1.0 else round(1.0, 2)\n        class_weight_log[key] = round(score_log, 2) if score_log &gt; 1.0 else round(1.0, 2)\n\n    return class_weight, class_weight_log\n\n# Class abundance for protein dataset\nlabels_dict = new_dict\n\nprint('\\nTrue class weights:')\nprint(create_class_weight(labels_dict)[0])\nprint('\\nLog-dampened class weights:')\nprint(create_class_weight(labels_dict)[1])\n```\n\nPass this class weights values to **model.fit()** function for the **class_weight** parameter.",
    "813497": "**Output for TRAINING_FILES+VAlIDATION_FILES combined,**\n\n**True class weights:**\n\n`{'iris': 16.3, 'wild rose': 17.17, 'wild geranium': 18.13, 'common dandelion': 22.65, 'rose': 27.72, 'sunflower': 27.72, 'daisy': 30.21, 'common tulip': 32.67, 'morning glory': 41.68, 'pink primrose': 46.91, 'spear thistle': 48.43, 'buttercup': 48.86, 'windflower': 49.0, 'yellow iris': 56.19, 'petunia': 63.57, 'wallflower': 74.17, 'water lily': 76.23, 'frangipani': 83.16, 'cyclamen ': 87.12, 'foxglove': 91.47, 'lotus': 91.47, 'azalea': 93.02, 'snapdragon': 93.55, 'hibiscus': 95.17, 'bougainvillea': 97.43, 'camellia': 100.4, 'marigold': 101.64, 'thorn apple': 102.27, 'magnolia': 106.23, 'passion flower': 106.92, 'artichoke': 107.61, 'anthurium': 108.32, 'primula': 111.25, 'columbine': 113.55, 'bee balm': 115.14, 'poinsettia': 115.95, 'sweet william': 116.77, 'hippeastrum ': 120.18, 'bird of paradise': 121.07, 'carnation': 121.07, 'wild pansy': 121.96, 'gazania': 122.87, 'clematis': 126.65, 'mallow': 127.64, 'watercress': 132.78, 'pincushion flower': 132.78, 'daffodil': 132.78, 'tree poppy': 136.07, 'black-eyed susan': 137.21, 'king protea': 138.36, 'balloon flower': 141.94, 'gaura': 143.17, 'monkshood': 147.01, 'tiger lily': 147.01, 'toad lily': 148.33, 'grape hyacinth': 151.06, 'globe thistle': 152.45, 'corn poppy': 153.88, 'lenten rose': 175.16, 'barberton daisy': 200.79, 'geranium': 203.27, 'sword lily': 203.27, 'pink-yellow dahlia': 219.53, 'bishop of llandaff': 219.53, 'cape flower': 222.5, 'californian poppy': 231.9, 'purple coneflower': 231.9, 'peruvian lily': 257.27, 'fritillary': 265.56, 'canna lily': 279.07, \"colt's foot\": 294.02, 'mexican petunia': 310.66, 'lilac hibiscus': 343.02, 'ruby-lipped cattleya': 350.32, 'orange dahlia': 357.93, 'bromelia': 374.2, 'stemless gentian': 374.2, 'osteospermum': 392.02, 'trumpet creeper': 411.62, 'desert-rose': 411.62, 'tree mallow': 433.29, 'japanese anemone': 445.0, 'silverbush': 457.36, 'cautleya spicata': 470.43, 'giant white arum lily': 484.26, 'great masterwort': 484.26, 'hard-leaved pocket orchid': 498.94, 'blackberry lily': 514.53, 'pink quill': 531.13, 'garden phlox': 531.13, 'blanket flower': 531.13, 'love in the mist': 548.83, 'spring crocus': 609.81, 'prince of wales feathers': 609.81, 'sweet pea': 609.81, 'globe-flower': 609.81, 'canterbury bells': 633.27, 'cosmos': 633.27, 'fire lily': 658.6, 'red ginger': 658.6, 'siam tulip': 686.04, 'bolero deep blue': 715.87, 'moon orchid': 715.87, 'alpine sea holly': 715.87}`\n\n**Log-dampened class weights:**\n\n`{'iris': 2.1, 'wild rose': 2.15, 'wild geranium': 2.2, 'common dandelion': 2.43, 'rose': 2.63, 'sunflower': 2.63, 'daisy': 2.72, 'common tulip': 2.79, 'morning glory': 3.04, 'pink primrose': 3.16, 'spear thistle': 3.19, 'buttercup': 3.2, 'windflower': 3.2, 'yellow iris': 3.34, 'petunia': 3.46, 'wallflower': 3.61, 'water lily': 3.64, 'frangipani': 3.73, 'cyclamen ': 3.77, 'foxglove': 3.82, 'lotus': 3.82, 'azalea': 3.84, 'snapdragon': 3.85, 'hibiscus': 3.86, 'bougainvillea': 3.89, 'camellia': 3.92, 'marigold': 3.93, 'thorn apple': 3.93, 'magnolia': 3.97, 'passion flower': 3.98, 'artichoke': 3.99, 'anthurium': 3.99, 'primula': 4.02, 'columbine': 4.04, 'bee balm': 4.05, 'poinsettia': 4.06, 'sweet william': 4.07, 'hippeastrum ': 4.1, 'bird of paradise': 4.1, 'carnation': 4.1, 'wild pansy': 4.11, 'gazania': 4.12, 'clematis': 4.15, 'mallow': 4.16, 'watercress': 4.2, 'pincushion flower': 4.2, 'daffodil': 4.2, 'tree poppy': 4.22, 'black-eyed susan': 4.23, 'king protea': 4.24, 'balloon flower': 4.26, 'gaura': 4.27, 'monkshood': 4.3, 'tiger lily': 4.3, 'toad lily': 4.31, 'grape hyacinth': 4.32, 'globe thistle': 4.33, 'corn poppy': 4.34, 'lenten rose': 4.47, 'barberton daisy': 4.61, 'geranium': 4.62, 'sword lily': 4.62, 'pink-yellow dahlia': 4.7, 'bishop of llandaff': 4.7, 'cape flower': 4.71, 'californian poppy': 4.75, 'purple coneflower': 4.75, 'peruvian lily': 4.86, 'fritillary': 4.89, 'canna lily': 4.94, \"colt's foot\": 4.99, 'mexican petunia': 5.05, 'lilac hibiscus': 5.14, 'ruby-lipped cattleya': 5.17, 'orange dahlia': 5.19, 'bromelia': 5.23, 'stemless gentian': 5.23, 'osteospermum': 5.28, 'trumpet creeper': 5.33, 'desert-rose': 5.33, 'tree mallow': 5.38, 'japanese anemone': 5.4, 'silverbush': 5.43, 'cautleya spicata': 5.46, 'giant white arum lily': 5.49, 'great masterwort': 5.49, 'hard-leaved pocket orchid': 5.52, 'blackberry lily': 5.55, 'pink quill': 5.58, 'garden phlox': 5.58, 'blanket flower': 5.58, 'love in the mist': 5.61, 'spring crocus': 5.72, 'prince of wales feathers': 5.72, 'sweet pea': 5.72, 'globe-flower': 5.72, 'canterbury bells': 5.76, 'cosmos': 5.76, 'fire lily': 5.8, 'red ginger': 5.8, 'siam tulip': 5.84, 'bolero deep blue': 5.88, 'moon orchid': 5.88, 'alpine sea holly': 5.88}`",
    "818485": "Did you test using it and what did you notice ? Improvement in score ? More training time ?"
  },
  "source": "meta"
}