{
  "id": 42989,
  "title": "PreProcessData",
  "url": "/competitions/intel-mobileodt-cervical-cancer-screening/discussion/42989",
  "author_name": "pkuer",
  "post_date": "2017-11-08T08:51:18.157000",
  "votes": 0,
  "comment_count": 0,
  "views": 0,
  "content": "<p>import numpy as np\nimport pandas as pd\nimport csv\nfrom subprocess import check_output\nprint(check_output(['ls','input']).decode('utf8'))</p>\n\n<h1>np.set_printoptions(threshold=np.inf)   #全部输出</h1>\n\n<p>import glob\nfrom PIL import ImageFilter, ImageStat, Image, ImageDraw\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport cv2</p>\n\n<h1>Pool类可以提供指定数量的进程供用户调用，当有新的请求提交到Pool中时，</h1>\n\n<h1>如果池还没有满，就会创建一个新的进程来执行请求。如果池满，请求就会告知先等待，</h1>\n\n<h1>直到池中有进程结束，才会创建新的进程来执行这些请求。</h1>\n\n<h1>multiprocessing包是Python中的多进程管理包,可以利用multiprocessing.Process对象来创建一个进程</h1>\n\n<p>from multiprocessing import Pool, cpu_count</p>\n\n<h1>并发的线程数,在multiprocessing模块中的cpu_count()函数已经实现了该功能</h1>\n\n<p>train1 = glob.glob(\"input/train/<em>/</em>.jpg\")\ntrain2 = glob.glob(\"input/train/Additional/*<em>/</em>.jpg\")\nprint(train1)\nprint(train2)</p>\n\n<p>train1 = pd.DataFrame([[p.split('/')[2],p.split('/')[3],p] for p in train1], columns = ['type','image','path'])\ntrain2 = pd.DataFrame([[p.split('/')[3],p.split('/')[4],p] for p in train2],columns = ['type','image','path'])\ntrain = train1.append(train2)</p>\n\n<h1>print(train1)</h1>\n\n<h1>print(train2)</h1>\n\n<p>print(train)</p>\n\n<h1>train2 = pd.DataFrame([p.split('/')[3]])</h1>\n\n<h1>train.to_csv('train.csv')</h1>\n\n<p>test = glob.glob(\"input/test/*.jpg\")</p>\n\n<h1>print(test)</h1>\n\n<p>test = pd.DataFrame([[p.split('/')[2],p] for p in test], columns = ['image','path'])\nprint(test)</p>\n\n<p>types = train.groupby('type', as_index=False).count()\ntypes.plot(kind='bar', x='type', y='path', figsize=(7,4))\nplt.show()</p>\n\n<h1>Pool类可以提供指定数量的进程供用户调用，当有新的请求提交到Pool中时，</h1>\n\n<h1>如果池还没有满，就会创建一个新的进程来执行请求。如果池满，请求就会告知先等待，</h1>\n\n<h1>直到池中有进程结束，才会创建新的进程来执行这些请求。</h1>\n\n<h1>multiprocessing包是Python中的多进程管理包,可以利用multiprocessing.Process对象来创建一个进程</h1>\n\n<h1>并发的线程数,在multiprocessing模块中的cpu_count()函数已经实现了该功能</h1>\n\n<p>def im_multi(path):\n    try:\n        im_stats_im_ = Image.open(path)\n        return [path, {'size': im_stats_im_.size}]\n    except:\n        print(path)\n        return [path, {'size': [0,0]}]</p>\n\n<h1>train=im_stats_df</h1>\n\n<p>def im_stats(im_stats_df):\n    im_stats_d = {}\n    p = Pool(cpu_count())\n    #从train里提取出路径传递给im_multi函数，函数执行后返回路径和图片的大小并且赋值给ret=[path, {'size': [0,0]}]\n    ret = p.map(im_multi, im_stats_df['path'])\n    for i in range(len(ret)):\n        #('input/train/Type_2/946.jpg', {'size': (2448, 3264)})\n        im_stats_d[ret[i][0]] = ret[i][1]</p>\n\n<pre><code>#print(im_stats_d.items())\n#按照X取出size并放入 im_stats_df中\nim_stats_df['size'] = im_stats_df['path'].map(lambda x: ' '.join(str(s) for s in im_stats_d[x]['size']))\n#print(im_stats_df)\nreturn im_stats_df\n</code></pre>\n\n<h1>对图片的大小进行改变</h1>\n\n<p>def get_im_cv2(path):\n    img = cv2.imread(path)\n    #线性插值法\n    resized = cv2.resize(img, (64, 64), cv2.INTER_LINEAR) #use cv2.resize(img, (64, 64), cv2.INTER_LINEAR)\n    return [path, resized]</p>\n\n<h1>对图片数据进行归一化操作</h1>\n\n<p>def normalize_image_features(paths):\n    imf_d = {}\n    p = Pool(cpu_count())\n    ret = p.map(get_im_cv2, paths)\n    #print(ret)\n    #['input/train/Type_2/1218.jpg', array([[[1, 1, 1],.....[0, 0, 0]]], dtype=uint8)]\n    for i in range(len(ret)):\n        imf_d[ret[i][0]] = ret[i][1]\n    #list转成dict\n    #print(imf_d)\n    ret = []\n    #imf_d.shape={'input/train/Type_2/1218.jpg' : array([[[1, 1, 1],.....[0, 0, 0]]], dtype=uint8)}\n    fdata = [imf_d[f] for f in paths]\n    fdata = np.array(fdata, dtype=np.uint8)\n    #(1481, 32, 32, 3)\n    #print(fdata.shape)\n    #矩阵的转置\n    fdata = fdata.transpose((0, 3, 1, 2))\n    #(1481, 3, 32, 32),3代表---&gt;RGB3通道\n    #print(fdata.shape)\n    fdata = fdata.astype('float32')\n    fdata = fdata / 255\n    return fdata</p>\n\n<h1>limit for Kaggle Demo</h1>\n\n<p>train = im_stats(train)</p>\n\n<h1>print(train)</h1>\n\n<h1>去除size为0的图片</h1>\n\n<p>train = train[train['size'] != '0 0'].reset_index(drop=True) #corrupt images removed\nprint(\"Bad images removed\")\nprint(\"loading train data\")</p>\n\n<h1>数据归一化</h1>\n\n<p>train_data = normalize_image_features(train['path'])\nprint(\"train data saved\")\nnp.save('train.npy', train_data, allow_pickle=True, fix_imports=True)</p>\n\n<p>from sklearn.preprocessing import LabelEncoder</p>\n\n<h1>LabelEncoder 是对不连续的数字或者文本进行编号</h1>\n\n<h1>([1,1,100,67,5])</h1>\n\n<h1>array([0,0,3,2,1])</h1>\n\n<p>le = LabelEncoder()\ntrain_target = le.fit_transform(train['type'].values)\nprint(le.classes_)\nprint(train_target)</p>\n\n<h1>['Type_1' 'Type_2' 'Type_3'],[1,0,2]</h1>\n\n<p>np.save('train_target.npy', train_target, allow_pickle=True, fix_imports=True)</p>\n\n<p>test_data = normalize_image_features(test['path'])</p>\n\n<h1>(512, 3, 32, 32)</h1>\n\n<h1>print(test_data.shape)</h1>\n\n<p>np.save('test.npy', test_data, allow_pickle=True, fix_imports=True)\nprint(\"test data loaded\")\ntest_id = test.image.values</p>\n\n<h1>print(test_id)#['67.jpg' '206.jpg' '357.jpg' '23.jpg'......</h1>\n\n<h1>从test中独处图片的序号</h1>\n\n<p>np.save('test_id.npy', test_id, allow_pickle=True, fix_imports=True)</p>",
  "messages": [
    {
      "id": 241194,
      "postDate": "2017-11-08T08:51:18.157Z",
      "content": "<p>import numpy as np\nimport pandas as pd\nimport csv\nfrom subprocess import check_output\nprint(check_output(['ls','input']).decode('utf8'))</p>\n\n<h1>np.set_printoptions(threshold=np.inf)   #全部输出</h1>\n\n<p>import glob\nfrom PIL import ImageFilter, ImageStat, Image, ImageDraw\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport cv2</p>\n\n<h1>Pool类可以提供指定数量的进程供用户调用，当有新的请求提交到Pool中时，</h1>\n\n<h1>如果池还没有满，就会创建一个新的进程来执行请求。如果池满，请求就会告知先等待，</h1>\n\n<h1>直到池中有进程结束，才会创建新的进程来执行这些请求。</h1>\n\n<h1>multiprocessing包是Python中的多进程管理包,可以利用multiprocessing.Process对象来创建一个进程</h1>\n\n<p>from multiprocessing import Pool, cpu_count</p>\n\n<h1>并发的线程数,在multiprocessing模块中的cpu_count()函数已经实现了该功能</h1>\n\n<p>train1 = glob.glob(\"input/train/<em>/</em>.jpg\")\ntrain2 = glob.glob(\"input/train/Additional/*<em>/</em>.jpg\")\nprint(train1)\nprint(train2)</p>\n\n<p>train1 = pd.DataFrame([[p.split('/')[2],p.split('/')[3],p] for p in train1], columns = ['type','image','path'])\ntrain2 = pd.DataFrame([[p.split('/')[3],p.split('/')[4],p] for p in train2],columns = ['type','image','path'])\ntrain = train1.append(train2)</p>\n\n<h1>print(train1)</h1>\n\n<h1>print(train2)</h1>\n\n<p>print(train)</p>\n\n<h1>train2 = pd.DataFrame([p.split('/')[3]])</h1>\n\n<h1>train.to_csv('train.csv')</h1>\n\n<p>test = glob.glob(\"input/test/*.jpg\")</p>\n\n<h1>print(test)</h1>\n\n<p>test = pd.DataFrame([[p.split('/')[2],p] for p in test], columns = ['image','path'])\nprint(test)</p>\n\n<p>types = train.groupby('type', as_index=False).count()\ntypes.plot(kind='bar', x='type', y='path', figsize=(7,4))\nplt.show()</p>\n\n<h1>Pool类可以提供指定数量的进程供用户调用，当有新的请求提交到Pool中时，</h1>\n\n<h1>如果池还没有满，就会创建一个新的进程来执行请求。如果池满，请求就会告知先等待，</h1>\n\n<h1>直到池中有进程结束，才会创建新的进程来执行这些请求。</h1>\n\n<h1>multiprocessing包是Python中的多进程管理包,可以利用multiprocessing.Process对象来创建一个进程</h1>\n\n<h1>并发的线程数,在multiprocessing模块中的cpu_count()函数已经实现了该功能</h1>\n\n<p>def im_multi(path):\n    try:\n        im_stats_im_ = Image.open(path)\n        return [path, {'size': im_stats_im_.size}]\n    except:\n        print(path)\n        return [path, {'size': [0,0]}]</p>\n\n<h1>train=im_stats_df</h1>\n\n<p>def im_stats(im_stats_df):\n    im_stats_d = {}\n    p = Pool(cpu_count())\n    #从train里提取出路径传递给im_multi函数，函数执行后返回路径和图片的大小并且赋值给ret=[path, {'size': [0,0]}]\n    ret = p.map(im_multi, im_stats_df['path'])\n    for i in range(len(ret)):\n        #('input/train/Type_2/946.jpg', {'size': (2448, 3264)})\n        im_stats_d[ret[i][0]] = ret[i][1]</p>\n\n<pre><code>#print(im_stats_d.items())\n#按照X取出size并放入 im_stats_df中\nim_stats_df['size'] = im_stats_df['path'].map(lambda x: ' '.join(str(s) for s in im_stats_d[x]['size']))\n#print(im_stats_df)\nreturn im_stats_df\n</code></pre>\n\n<h1>对图片的大小进行改变</h1>\n\n<p>def get_im_cv2(path):\n    img = cv2.imread(path)\n    #线性插值法\n    resized = cv2.resize(img, (64, 64), cv2.INTER_LINEAR) #use cv2.resize(img, (64, 64), cv2.INTER_LINEAR)\n    return [path, resized]</p>\n\n<h1>对图片数据进行归一化操作</h1>\n\n<p>def normalize_image_features(paths):\n    imf_d = {}\n    p = Pool(cpu_count())\n    ret = p.map(get_im_cv2, paths)\n    #print(ret)\n    #['input/train/Type_2/1218.jpg', array([[[1, 1, 1],.....[0, 0, 0]]], dtype=uint8)]\n    for i in range(len(ret)):\n        imf_d[ret[i][0]] = ret[i][1]\n    #list转成dict\n    #print(imf_d)\n    ret = []\n    #imf_d.shape={'input/train/Type_2/1218.jpg' : array([[[1, 1, 1],.....[0, 0, 0]]], dtype=uint8)}\n    fdata = [imf_d[f] for f in paths]\n    fdata = np.array(fdata, dtype=np.uint8)\n    #(1481, 32, 32, 3)\n    #print(fdata.shape)\n    #矩阵的转置\n    fdata = fdata.transpose((0, 3, 1, 2))\n    #(1481, 3, 32, 32),3代表---&gt;RGB3通道\n    #print(fdata.shape)\n    fdata = fdata.astype('float32')\n    fdata = fdata / 255\n    return fdata</p>\n\n<h1>limit for Kaggle Demo</h1>\n\n<p>train = im_stats(train)</p>\n\n<h1>print(train)</h1>\n\n<h1>去除size为0的图片</h1>\n\n<p>train = train[train['size'] != '0 0'].reset_index(drop=True) #corrupt images removed\nprint(\"Bad images removed\")\nprint(\"loading train data\")</p>\n\n<h1>数据归一化</h1>\n\n<p>train_data = normalize_image_features(train['path'])\nprint(\"train data saved\")\nnp.save('train.npy', train_data, allow_pickle=True, fix_imports=True)</p>\n\n<p>from sklearn.preprocessing import LabelEncoder</p>\n\n<h1>LabelEncoder 是对不连续的数字或者文本进行编号</h1>\n\n<h1>([1,1,100,67,5])</h1>\n\n<h1>array([0,0,3,2,1])</h1>\n\n<p>le = LabelEncoder()\ntrain_target = le.fit_transform(train['type'].values)\nprint(le.classes_)\nprint(train_target)</p>\n\n<h1>['Type_1' 'Type_2' 'Type_3'],[1,0,2]</h1>\n\n<p>np.save('train_target.npy', train_target, allow_pickle=True, fix_imports=True)</p>\n\n<p>test_data = normalize_image_features(test['path'])</p>\n\n<h1>(512, 3, 32, 32)</h1>\n\n<h1>print(test_data.shape)</h1>\n\n<p>np.save('test.npy', test_data, allow_pickle=True, fix_imports=True)\nprint(\"test data loaded\")\ntest_id = test.image.values</p>\n\n<h1>print(test_id)#['67.jpg' '206.jpg' '357.jpg' '23.jpg'......</h1>\n\n<h1>从test中独处图片的序号</h1>\n\n<p>np.save('test_id.npy', test_id, allow_pickle=True, fix_imports=True)</p>",
      "rawMarkdown": "import numpy as np\nimport pandas as pd\nimport csv\nfrom subprocess import check_output\nprint(check_output(['ls','input']).decode('utf8'))\n#np.set_printoptions(threshold=np.inf)   #全部输出\nimport glob\nfrom PIL import ImageFilter, ImageStat, Image, ImageDraw\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport cv2\n#Pool类可以提供指定数量的进程供用户调用，当有新的请求提交到Pool中时，\n# 如果池还没有满，就会创建一个新的进程来执行请求。如果池满，请求就会告知先等待，\n# 直到池中有进程结束，才会创建新的进程来执行这些请求。\n#multiprocessing包是Python中的多进程管理包,可以利用multiprocessing.Process对象来创建一个进程\nfrom multiprocessing import Pool, cpu_count\n#并发的线程数,在multiprocessing模块中的cpu_count()函数已经实现了该功能\n\ntrain1 = glob.glob(\"input/train/*/*.jpg\")\ntrain2 = glob.glob(\"input/train/Additional/**/*.jpg\")\nprint(train1)\nprint(train2)\n\ntrain1 = pd.DataFrame([[p.split('/')[2],p.split('/')[3],p] for p in train1], columns = ['type','image','path'])\ntrain2 = pd.DataFrame([[p.split('/')[3],p.split('/')[4],p] for p in train2],columns = ['type','image','path'])\ntrain = train1.append(train2)\n#print(train1)\n#print(train2)\nprint(train)\n#train2 = pd.DataFrame([p.split('/')[3]])\n#train.to_csv('train.csv')\ntest = glob.glob(\"input/test/*.jpg\")\n#print(test)\ntest = pd.DataFrame([[p.split('/')[2],p] for p in test], columns = ['image','path'])\nprint(test)\n\n\ntypes = train.groupby('type', as_index=False).count()\ntypes.plot(kind='bar', x='type', y='path', figsize=(7,4))\nplt.show()\n\n\n#Pool类可以提供指定数量的进程供用户调用，当有新的请求提交到Pool中时，\n# 如果池还没有满，就会创建一个新的进程来执行请求。如果池满，请求就会告知先等待，\n# 直到池中有进程结束，才会创建新的进程来执行这些请求。\n#multiprocessing包是Python中的多进程管理包,可以利用multiprocessing.Process对象来创建一个进程\n#并发的线程数,在multiprocessing模块中的cpu_count()函数已经实现了该功能\ndef im_multi(path):\n    try:\n        im_stats_im_ = Image.open(path)\n        return [path, {'size': im_stats_im_.size}]\n    except:\n        print(path)\n        return [path, {'size': [0,0]}]\n\n#train=im_stats_df\ndef im_stats(im_stats_df):\n    im_stats_d = {}\n    p = Pool(cpu_count())\n    #从train里提取出路径传递给im_multi函数，函数执行后返回路径和图片的大小并且赋值给ret=[path, {'size': [0,0]}]\n    ret = p.map(im_multi, im_stats_df['path'])\n    for i in range(len(ret)):\n        #('input/train/Type_2/946.jpg', {'size': (2448, 3264)})\n        im_stats_d[ret[i][0]] = ret[i][1]\n\n    #print(im_stats_d.items())\n    #按照X取出size并放入 im_stats_df中\n    im_stats_df['size'] = im_stats_df['path'].map(lambda x: ' '.join(str(s) for s in im_stats_d[x]['size']))\n    #print(im_stats_df)\n    return im_stats_df\n#对图片的大小进行改变\ndef get_im_cv2(path):\n    img = cv2.imread(path)\n    #线性插值法\n    resized = cv2.resize(img, (64, 64), cv2.INTER_LINEAR) #use cv2.resize(img, (64, 64), cv2.INTER_LINEAR)\n    return [path, resized]\n#对图片数据进行归一化操作\ndef normalize_image_features(paths):\n    imf_d = {}\n    p = Pool(cpu_count())\n    ret = p.map(get_im_cv2, paths)\n    #print(ret)\n    #['input/train/Type_2/1218.jpg', array([[[1, 1, 1],.....[0, 0, 0]]], dtype=uint8)]\n    for i in range(len(ret)):\n        imf_d[ret[i][0]] = ret[i][1]\n    #list转成dict\n    #print(imf_d)\n    ret = []\n    #imf_d.shape={'input/train/Type_2/1218.jpg' : array([[[1, 1, 1],.....[0, 0, 0]]], dtype=uint8)}\n    fdata = [imf_d[f] for f in paths]\n    fdata = np.array(fdata, dtype=np.uint8)\n    #(1481, 32, 32, 3)\n    #print(fdata.shape)\n    #矩阵的转置\n    fdata = fdata.transpose((0, 3, 1, 2))\n    #(1481, 3, 32, 32),3代表---&gt;RGB3通道\n    #print(fdata.shape)\n    fdata = fdata.astype('float32')\n    fdata = fdata / 255\n    return fdata\n#limit for Kaggle Demo\ntrain = im_stats(train)\n#print(train)\n#去除size为0的图片\ntrain = train[train['size'] != '0 0'].reset_index(drop=True) #corrupt images removed\nprint(\"Bad images removed\")\nprint(\"loading train data\")\n#数据归一化\ntrain_data = normalize_image_features(train['path'])\nprint(\"train data saved\")\nnp.save('train.npy', train_data, allow_pickle=True, fix_imports=True)\n\n\nfrom sklearn.preprocessing import LabelEncoder\n#LabelEncoder 是对不连续的数字或者文本进行编号\n#([1,1,100,67,5])\n#array([0,0,3,2,1])\nle = LabelEncoder()\ntrain_target = le.fit_transform(train['type'].values)\nprint(le.classes_)\nprint(train_target)\n#['Type_1' 'Type_2' 'Type_3'],[1,0,2]\nnp.save('train_target.npy', train_target, allow_pickle=True, fix_imports=True)\n\n\ntest_data = normalize_image_features(test['path'])\n#(512, 3, 32, 32)\n#print(test_data.shape)\nnp.save('test.npy', test_data, allow_pickle=True, fix_imports=True)\nprint(\"test data loaded\")\ntest_id = test.image.values\n#print(test_id)#['67.jpg' '206.jpg' '357.jpg' '23.jpg'......\n#从test中独处图片的序号\nnp.save('test_id.npy', test_id, allow_pickle=True, fix_imports=True)"
    }
  ],
  "comments": [],
  "raw_markdown_by_id": {
    "241194": "import numpy as np\nimport pandas as pd\nimport csv\nfrom subprocess import check_output\nprint(check_output(['ls','input']).decode('utf8'))\n#np.set_printoptions(threshold=np.inf)   #全部输出\nimport glob\nfrom PIL import ImageFilter, ImageStat, Image, ImageDraw\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport cv2\n#Pool类可以提供指定数量的进程供用户调用，当有新的请求提交到Pool中时，\n# 如果池还没有满，就会创建一个新的进程来执行请求。如果池满，请求就会告知先等待，\n# 直到池中有进程结束，才会创建新的进程来执行这些请求。\n#multiprocessing包是Python中的多进程管理包,可以利用multiprocessing.Process对象来创建一个进程\nfrom multiprocessing import Pool, cpu_count\n#并发的线程数,在multiprocessing模块中的cpu_count()函数已经实现了该功能\n\ntrain1 = glob.glob(\"input/train/*/*.jpg\")\ntrain2 = glob.glob(\"input/train/Additional/**/*.jpg\")\nprint(train1)\nprint(train2)\n\ntrain1 = pd.DataFrame([[p.split('/')[2],p.split('/')[3],p] for p in train1], columns = ['type','image','path'])\ntrain2 = pd.DataFrame([[p.split('/')[3],p.split('/')[4],p] for p in train2],columns = ['type','image','path'])\ntrain = train1.append(train2)\n#print(train1)\n#print(train2)\nprint(train)\n#train2 = pd.DataFrame([p.split('/')[3]])\n#train.to_csv('train.csv')\ntest = glob.glob(\"input/test/*.jpg\")\n#print(test)\ntest = pd.DataFrame([[p.split('/')[2],p] for p in test], columns = ['image','path'])\nprint(test)\n\n\ntypes = train.groupby('type', as_index=False).count()\ntypes.plot(kind='bar', x='type', y='path', figsize=(7,4))\nplt.show()\n\n\n#Pool类可以提供指定数量的进程供用户调用，当有新的请求提交到Pool中时，\n# 如果池还没有满，就会创建一个新的进程来执行请求。如果池满，请求就会告知先等待，\n# 直到池中有进程结束，才会创建新的进程来执行这些请求。\n#multiprocessing包是Python中的多进程管理包,可以利用multiprocessing.Process对象来创建一个进程\n#并发的线程数,在multiprocessing模块中的cpu_count()函数已经实现了该功能\ndef im_multi(path):\n    try:\n        im_stats_im_ = Image.open(path)\n        return [path, {'size': im_stats_im_.size}]\n    except:\n        print(path)\n        return [path, {'size': [0,0]}]\n\n#train=im_stats_df\ndef im_stats(im_stats_df):\n    im_stats_d = {}\n    p = Pool(cpu_count())\n    #从train里提取出路径传递给im_multi函数，函数执行后返回路径和图片的大小并且赋值给ret=[path, {'size': [0,0]}]\n    ret = p.map(im_multi, im_stats_df['path'])\n    for i in range(len(ret)):\n        #('input/train/Type_2/946.jpg', {'size': (2448, 3264)})\n        im_stats_d[ret[i][0]] = ret[i][1]\n\n    #print(im_stats_d.items())\n    #按照X取出size并放入 im_stats_df中\n    im_stats_df['size'] = im_stats_df['path'].map(lambda x: ' '.join(str(s) for s in im_stats_d[x]['size']))\n    #print(im_stats_df)\n    return im_stats_df\n#对图片的大小进行改变\ndef get_im_cv2(path):\n    img = cv2.imread(path)\n    #线性插值法\n    resized = cv2.resize(img, (64, 64), cv2.INTER_LINEAR) #use cv2.resize(img, (64, 64), cv2.INTER_LINEAR)\n    return [path, resized]\n#对图片数据进行归一化操作\ndef normalize_image_features(paths):\n    imf_d = {}\n    p = Pool(cpu_count())\n    ret = p.map(get_im_cv2, paths)\n    #print(ret)\n    #['input/train/Type_2/1218.jpg', array([[[1, 1, 1],.....[0, 0, 0]]], dtype=uint8)]\n    for i in range(len(ret)):\n        imf_d[ret[i][0]] = ret[i][1]\n    #list转成dict\n    #print(imf_d)\n    ret = []\n    #imf_d.shape={'input/train/Type_2/1218.jpg' : array([[[1, 1, 1],.....[0, 0, 0]]], dtype=uint8)}\n    fdata = [imf_d[f] for f in paths]\n    fdata = np.array(fdata, dtype=np.uint8)\n    #(1481, 32, 32, 3)\n    #print(fdata.shape)\n    #矩阵的转置\n    fdata = fdata.transpose((0, 3, 1, 2))\n    #(1481, 3, 32, 32),3代表---&gt;RGB3通道\n    #print(fdata.shape)\n    fdata = fdata.astype('float32')\n    fdata = fdata / 255\n    return fdata\n#limit for Kaggle Demo\ntrain = im_stats(train)\n#print(train)\n#去除size为0的图片\ntrain = train[train['size'] != '0 0'].reset_index(drop=True) #corrupt images removed\nprint(\"Bad images removed\")\nprint(\"loading train data\")\n#数据归一化\ntrain_data = normalize_image_features(train['path'])\nprint(\"train data saved\")\nnp.save('train.npy', train_data, allow_pickle=True, fix_imports=True)\n\n\nfrom sklearn.preprocessing import LabelEncoder\n#LabelEncoder 是对不连续的数字或者文本进行编号\n#([1,1,100,67,5])\n#array([0,0,3,2,1])\nle = LabelEncoder()\ntrain_target = le.fit_transform(train['type'].values)\nprint(le.classes_)\nprint(train_target)\n#['Type_1' 'Type_2' 'Type_3'],[1,0,2]\nnp.save('train_target.npy', train_target, allow_pickle=True, fix_imports=True)\n\n\ntest_data = normalize_image_features(test['path'])\n#(512, 3, 32, 32)\n#print(test_data.shape)\nnp.save('test.npy', test_data, allow_pickle=True, fix_imports=True)\nprint(\"test data loaded\")\ntest_id = test.image.values\n#print(test_id)#['67.jpg' '206.jpg' '357.jpg' '23.jpg'......\n#从test中独处图片的序号\nnp.save('test_id.npy', test_id, allow_pickle=True, fix_imports=True)"
  }
}