{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Got a new job so stopped working on this\n\nI took a different angle if it helps anyone:\nFind strongest point in 5 second sample\nNormalize all arrays to have the max peak at the same point (to ease comparison)\nfocus on +/- 0.5 sec around that peak\nFingerprint the frequencies leading up to and after the peak and the intensities.\n\nI was looking to identify the region of birds for each of the 3 areas (say N. California), which would then limit the range of possible birds to choose from.\n\nKey differences I noticed in the data:\nslope leading to peak and away tended to be relatively unique\nfrequencies on average for the peak can be used to reduce the possible birds\nI normalized the frequencies measured to be in the range 1800-3600 by multipllying and dividing, as they would have resonances in that range and again it simplified comparing them.\n\n(then I got a new job and abandoned for the last month...)\n\nMaybe this will help give some new ideas, Mark\n\n==================\n\nOther features:\nThreading and Que Operations\nUsing sample data for submissions\nLow Pass and High Pass filter functions\nFFT with graphing\nNorth America Only function (as that is what 3 regions are specified in)\n\n","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Hot Ideas\n\n1.  Fingerprint, ideal steps\n    * Look for Max Magnitude (power) in first 3 min of song\n    * Start analysis say 1/10 second earlier if possible\n    * Find peak freqencies from there in steps as fingerprint (having way of capturing delay gaps)\n    * FP may look like: F: 1900,2015,1890, 1700, ..10, FP: 1900,3,x,2015,3,x...etc.\n    * Then scan at lower power for second bird if it exists, or bandgap our primary frequency of bird 1 and rebuild song backwards\n2.  Try getting all the data with only gaps and no FFT, ideally with more clever gap measurements (with a peak detection sweep...)\n3. Need to have at least 2/3 models, one binary to identify song/nocall , then maybe frequency buckets by freq class, then birds\n\n###### Version History, zoBirdFinder5A\n* v1 - Moved from v7B of same, Intending to be new data creation, with B of same as Submit version\n\n","execution_count":null},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"## Kudos to the help from the community:\n## some initial layout from:\n# https://www.kaggle.com/muhakabartay/birdcall-eda-basemap-geo-3d-elevation-submission/\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Imports","execution_count":null},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"import numpy as np \nimport pandas as pd \n\nimport os\n\nimport IPython.display as ipd\npd.set_option('max_columns', 35)\nimport matplotlib.pyplot as plt\n%matplotlib inline\nimport seaborn as sns\nimport pickle # serializing models  \n# save model\n# pickle.dump(model, open(filename, 'wb'))\n# load the model from disk\n# loaded_model = pickle.load(open(filename, 'rb'))\n# result = loaded_model.score(X_test, Y_test)\n\nimport shutil\n# shutil.copy2('/src/dir/file.ext', '/dst/dir/newname.ext') # complete target filename given\n# shutil.copy2('/src/file.ext', '/dst/dir') # target filename is /dst/dir/file.ext","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Imports\n\n\nimport matplotlib.image as mpimg\nfrom matplotlib.offsetbox import AnnotationBbox, OffsetImage\n\n# Map 1 library\nimport plotly.express as px\n\n# Map 2 libraries\nimport descartes\nimport geopandas as gpd\nfrom shapely.geometry import Point, Polygon\n\n#install numba==0.48\n# above is hack to not hit 0.5 removal of decorators\n\nimport numba\n#==0.48\nimport numba.decorators\n\nimport sklearn\n\nimport warnings\nwarnings.filterwarnings('ignore')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Audio / Threading Imports","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"import librosa\nimport librosa.display\nfrom joblib import Parallel, delayed, cpu_count\nimport time, math # for display type use ME\n\n\nimport pylab\n\nimport torchaudio\nfrom scipy import signal\n\nimport gc\nfrom path import Path","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import threading, queue\n\n# q = queue.Queue()\nbRUNNING = True","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"bUseNoiseReduce = False\nif bUseNoiseReduce == True:\n## Imports for testing\n    !pip install noisereduce\n\n    import noisereduce as nr # noise reduction","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"bSUBMITRUN = False\nbPREDICTRUN = False ## Set to False for Data, True for Predict runs (automatically)\nbFASTRUN = True\nbRUNPARALLEL = True\n\nnSAMPLERATE = 22050\nbFILTER = True # butter filter\nnMAXSONGITER = 5","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Multi-Threading Update\n\n\n\n","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## Data Files","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"\n\nmodel_filename = 'BFinder_model.sav'  # if first run, copy file to output folder (for path consistency..)\nif os.path.isfile(model_filename):\n    print(\"Model File already exists in output.\")\nelse:\n    shutil.copy2('/kaggle/input/zobirdsongdata3/BFinder_model.sav', 'BFinder_model.sav') # target filename is /dst/dir/file.ext\n    print(\"Copying from saved model to output folder\")\n# Step on file in output if not saved\n\nTEST = Path(\"../input/birdsong-recognition/test_audio\").exists()\nbSUBMITRUN = TEST\nif TEST:\n    DATA_DIR = str(Path(\"../input/birdsong-recognition\"))\n    print(\"SUBMIT PATH EXISTS\")\nelse:\n    # dataset created by @shonenkov, thanks!\n    DATA_DIR = str(Path(\"../input/birdcall-check\"))\n    print(\"NON-SUBMIT - Using Alternative PATH\")\n\nprint(\"DATA_DIR (Dynamic) = \",DATA_DIR)\n\n\nTEST_FOLDER = DATA_DIR + '/test_audio/'\ntest = pd.read_csv(DATA_DIR + \"/test.csv\")\n\ntest.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def onlyNAmerica(data, data_info=[]): # top 3 are N America, will filter out of data, based on data_info (in case different)\n    if len(data_info)<1:\n        data_info = data\n    top_3 = list(data_info['country'].value_counts().head(3).reset_index()['index'])\n    dInfo_top_3 = data_info[data_info['country'].isin(top_3)]\n    data_top_3 = data[data['filename'].isin(dInfo_top_3['filename']) ]  #logical_not = data[~data\n    return data_top_3","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"TRAIN_DIR = str(Path(\"../input/birdsong-recognition\"))\ntrain = pd.read_csv(TRAIN_DIR + \"/train.csv\")\nsub = pd.read_csv(TRAIN_DIR + \"/sample_submission.csv\")\n\n# #######  train = onlyNAmerica(train)\n\n\n## OUPUT FILES\ntrain_part1 = pd.read_csv('/kaggle/input/zobirdsongdata2/Bproj_data0801-0_10.csv')\ntrain_part2 = pd.read_csv('/kaggle/input/zobirdsongdata2/Bproj_data0801-10_15.csv')\ntrain_part3 = pd.read_csv('/kaggle/input/zobirdsongdata2/Bproj_data0801-15_25.csv')\ntrain_part4 = pd.read_csv('/kaggle/input/zobirdsongdata2/Bproj_data0801-25_40.csv')\ntrain_part5 = pd.read_csv('/kaggle/input/zobirdsongdata2/Bproj_data0801-40_55.csv')\ntrain_part6 = pd.read_csv('/kaggle/input/zobirdsongdata2/Bproj_data0801-55_73.csv')\ntrain_part7 = pd.read_csv('/kaggle/input/zobirdsongdata2/Bproj_data0801-73_85.csv')\ntrain_part8 = pd.read_csv('/kaggle/input/zobirdsongdata2/Bproj_data0801-85_95.csv')\ntrain_part9 = pd.read_csv('/kaggle/input/zobirdsongdata2/Bproj_data0801-95_100.csv')\n#../input/zobirdsongdata2/Bproj_data0801-0_10.csv\ntrain_part1b = pd.read_csv('/kaggle/input/zobirdsongdata3/Bproj_data_0_4.csv')\ntrain_part2b = pd.read_csv('/kaggle/input/zobirdsongdata3/Bproj_data_5_20.csv')\ntrain_part3b = pd.read_csv('/kaggle/input/zobirdsongdata3/Bproj_data_20_35.csv')\ntrain_part4b = pd.read_csv('/kaggle/input/zobirdsongdata3/Bproj_data_35_50.csv')\ntrain_part5b = pd.read_csv('/kaggle/input/zobirdsongdata3/Bproj_data_50_65.csv')\ntrain_part6b = pd.read_csv('/kaggle/input/zobirdsongdata3/Bproj_data_65_80.csv')\ntrain_part7b = pd.read_csv('/kaggle/input/zobirdsongdata3/Bproj_data_80_95.csv')\n\nprint(\"Shape train: \", train.shape)\nprint(\"Shape  test: \", test.shape)\nprint(\"Shape   sub: \", sub.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"\n# def addPathToTest(test, root_path=TEST_FOLDER):\n#     test['path'] = root_path + test['audio_id'] + '.mp3'      \n#     return test\n    \n# test_path = addPathToTest(test)\n\n# test_path.head()\n# urlSong = test_path.loc[0,'path']\n# sound_clip = load_test_clip(urlSong,0)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Create Q Rows Code\n============================================================================\n\n","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"\n\nbPREDICTRUN = True\n\ndef trainCreateQrow(filename, duration=None, bird=''):\n    dfReturn = pd.DataFrame([['', '', 0, '']],\\\n                   columns=['site', 'row_id', 'seconds', 'audio_id'])\n    \n    if len(filename)>5:\n        head, tail = os.path.split(filename)\n        file = os.path.splitext(os.path.basename(filename))[0] # no extension \n    else:\n        file = \"NA\"\n        \n    dfReturn['audio_id'] = file\n    dfReturn['row_id'] = file\n    dfReturn['seconds'] = None\n    dfReturn['site'] = 'site_3' ## anything mostly\n    print(filename)\n    return createQrow(dfReturn, 0, fSecFile=None, root_path=head+'/' )\n\n\n\ndef createQrow(test, nIndex, fSecFile=None, root_path=TEST_FOLDER):\n    bIterate = True\n    dfReturn = pd.DataFrame()\n    rowTest = test.iloc[int(nIndex)]\n    rowTest['path'] = root_path + rowTest['audio_id'] + '.mp3'\n\n#     fSecFile=11 ### HACK for testing\n    if (rowTest['site'] == 'site_1') or (rowTest['site'] == 'site_2'):\n        rowTest['nIter'] = (rowTest['seconds'] /5)-1\n        dfReturn = dfReturn.append(rowTest, ignore_index=True)\n#     elif fSecFile == None:\n#         ## ERROR\n#         print(\"ERROR, Site 3 has not Duraton passed in.\")\n#         rowTest['nIter'] = -1\n    else:\n        if fSecFile == None:\n            print(\"Loading....\")\n            fSecFile = librosa.get_duration(filename=rowTest['path'])\n            print(\"done, fSecFile = \", fSecFile)\n#         print(\"Site 3 has not Duraton passed in.\")\n        secIteration =  5.0\n        nTotIter = math.ceil(fSecFile / secIteration)\n        if bPREDICTRUN == False:\n            if bIterate == True:\n                secLastIter = fSecFile % secIteration  \n                if secLastIter < 1.5 and fSecFile > secIteration:\n                    nTotIter = nTotIter-1 # Last iter too short, let's skip it  ### Keep if Submission file??\n            else:\n                nTotIter = 1\n            if nTotIter > nMAXSONGITER and bPREDICTRUN == False:\n                nTotIter = nMAXSONGITER # limit to this number of segments       \n        secEnd = secIteration       \n        for nIter in range(nTotIter):\n            if fSecFile <= secEnd:\n                secEnd = fSecFile  \n            rowTest['nIter'] = nIter\n            rowTest['seconds'] = (nIter+1)*5.0 ## endTime\n            secStart = secEnd\n            secEnd = secEnd + secIteration\n            dfReturn = dfReturn.append(rowTest, ignore_index=True)\n            \n    nRetSize = len(dfReturn)\n    return dfReturn, nRetSize\n\n\n\n\n# dfTest = pd.DataFrame()\n# for rowID in range(len(test)):\n#     dfTest = dfTest.append(createQrow(test,rowID), ignore_index=True)\n\n    \n# dfTest[-30:]   \n\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sampleBird_file1 = '/kaggle/input/birdsong-recognition/train_audio/astfly/XC122918.mp3'\n\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"\n# trainCreateQrow('07ab324c602e4afab65ddbcc746c31b5.mp3', None) #\n# trainCreateQrow('41e6fe6504a34bf6846938ba78d13df1.mp3', None) #sampleBird_file1","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"### N. America Filtered data\ntrainNA = onlyNAmerica(train)\nprint(\"Filter our migration data:  All, N.America = \",len(train), len(trainNA))\ndbNAEbirdCodes = trainNA['ebird_code'].unique()\ndbNAEbirdCodes=np.append(dbNAEbirdCodes,'nocall')\ndbNAEbirdCodes=np.append(dbNAEbirdCodes,'unknown')\nnEbirdCodesNA = len(dbNAEbirdCodes) # same birds, migration data is skipped however","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"\ndbEbirdCodes = train['ebird_code'].unique()\ndbEbirdCodes=np.append(dbEbirdCodes,'nocall')\ndbEbirdCodes=np.append(dbEbirdCodes,'unknown')\nnEbirdCodes = len(dbEbirdCodes)\nenumECodes = dict(zip(dbEbirdCodes,range(nEbirdCodes)))  # enumECodes['wooscj2'] -> =255\nenumECodesRev = dict(zip(range(nEbirdCodes),dbEbirdCodes))\n\nprint(\"ECodes = \", nEbirdCodes,\"total, \", dict(list(enumECodes.items())[-20:]))\nprint(\"  ##  \")\nprint(\"enumECodesRev = \", nEbirdCodes,\"total, \", dict(list(enumECodesRev.items())[-20:]))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def countTrainBirds(data):\n    print(\"Shape train: \", data.shape)\n    dbTrainBirds = data['ecode'].unique()\n    print(\"Training set has \", len(dbTrainBirds), \" unique birds in it.\")\n\n    dbMissingEBirds = []\n    dbMissingBirds = []\n    for bird in range(nEbirdCodes):\n        if bird not in dbTrainBirds:\n            dbMissingEBirds = np.append(dbMissingEBirds, bird)\n            dbMissingBirds = np.append(dbMissingBirds, enumECodesRev[bird])\n    print(\"Training set Missing ecodes =  \",dbMissingEBirds)\n    print(\"Training set Missing birds =  \",dbMissingBirds)\n    return len(dbMissingEBirds)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"\nbird_columns = ['nSongID', 'ecode','file', 'nUpRows', 'nDownRows','10',\\\n                'nCrossings','bArea', 'sNotes', 'sLength', 'sRange', 'sTrendline', 'sPitch', 'sSpacing', 'sDutyCycle',\\\n               ]\nbird_dbRow = [0,'None','None',0, 0,0,\\\n             0, 'None', 0, 0.0, 0.0, 0.0, 0.0,0.0, 0.0, \\\n             ]\n\nbirdRow = pd.Series(data=bird_dbRow, index=bird_columns)\nbirdDb = pd.DataFrame(columns = bird_columns)\nnewBirdRow = birdRow.copy()\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"## USED\nbFinder_columns = ['nSongID', 'ecode','file_nIt', 'primFr', 'avgFr','primSl','dCycle', 'nSegm','fFundFreq','nFFmult', \\\n                 'f1', 'f2','f3','f4', 'f5','f6','f7','f8', 'f9','f10',\\\n                 'fp1', 'fp2','fp3','fp4', 'fp5','fp6','fp7','fp8', 'fp9','fp10',\\\n               ]\nbFinder_row = [0,0,'None',0.0, 0.0,0.0, 0.0,0.0, 0.0,0.0, \\\n             0.0,0.0, 0.0,0.0, 0.0,0.0, 0.0,0.0, 0.0,0.0, \\\n             0.0,0.0, 0.0,0.0, 0.0,0.0, 0.0,0.0, 0.0,0.0,\\\n             ]\n\nbFinderRow = pd.Series(data=bFinder_row, index=bFinder_columns)\nbFinderDb = pd.DataFrame(columns = bFinder_columns)\nnewBFinderRow = bFinderRow.copy()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Setting up Que's\n\nq = queue.Queue()","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Que for songs waiting processing, labelled as done afterwards\nqueToProcess_columns = ['nIndex', 'Species', 'ecode','file', 'nBirdRefId', 'nStatus','startSec', 'duration' ]\nqueToProcess_row = [0, 'None', 'None','None', 0, 0, 0.0, 0.0 ]\nqueToProcess = pd.DataFrame(columns = queToProcess_columns)\n\nqToProcess_columns = ['nIndex','file', 'fileRefId', 'nStatus' ,'startSec', 'duration', 'nIter', 'nLoc']\nqToProcess_row = [0, 'None', 'None', 0 , 0.0, 0.0, 0,0 ]\nnewToProcessRow = pd.Series(data=qToProcess_row, index=qToProcess_columns)\nqToProcess = queue.Queue(maxsize=6) # maxsize unlimited.. 3 worked\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Main Starting functions","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def startThreads():\n            \n    # turn-on the worker thread\n    threading.Thread(target=workerFileQ, daemon=True).start()\n\n# if __name__ == '__main__':\n#     main()\n    return\n\n\ndef initRun():\n    initDb()\n    nPipeRuns = 0\n    nFFTGraphs = 0\n    return\n\ndef initDb():\n    global birdDb\n    global bFinderDb\n    global queToProcess\n#     global queSongs\n    global queSongInfo\n    global bRUNNING\n    birdDb.drop(birdDb.index, inplace=True) # flush db of rows\n    bFinderDb.drop(birdDb.index, inplace=True) # flush db of rows\n    queToProcess.drop(queToProcess.index, inplace=True) # flush db of rows\n#     queSongs.drop(queSongs.index, inplace=True) # flush db of rows\n#     queSongInfo.drop(queSongInfo.index, inplace=True) # flush db of rows\n    bRUNNING = False\n    time.sleep(4) # Allow child threads to stop\n    bRUNNING = False\n    # Start threads...\n    startThreads()\n    return\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Main support functions","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"\n\ndef newRow():\n    global newBirdRow\n    newBirdRow = birdRow.copy()\n    return\ndef newBRow():\n    global newBFinderRow\n    newBFinderRow = bFinderRow.copy()\n    return\n\ndef addRowValue(paramName = \"None\", paramValue = \"0.0\"):\n    global newBirdRow\n    newBirdRow[paramName] = paramValue\n    return\ndef addBRowValue(paramName = \"None\", paramValue = \"0.0\"):\n    global newBFinderRow\n    newBFinderRow[paramName] = paramValue\n    return\n\ndef appendRow():  \n    global birdDb\n    global newBirdRow \n    birdDb = birdDb.append(newBirdRow, ignore_index=True)\n    nIndex =  len(birdDb.index)-1\n    return nIndex  # return index of new row\ndef appendBRow():  \n    global bFinderDb\n    global newBFinderRow \n    bFinderDb = bFinderDb.append(newBFinderRow, ignore_index=True)\n    nIndex =  len(bFinderDb.index)-1\n    return nIndex  # return index of new row\n\n\ndef getBirdRow( nIndex = -1, lSpecies = \"None\", nSpecies = -1):\n    global birdDb\n    global newBirdRow\n    if nIndex >= 0 and nIndex in birdDb.index: # if valid\n        lSpecies = birdDb.at[nIndex,'Species']\n        nSpecies = birdDb.at[nIndex,'nSpecies']\n    else:\n        if nSpecies < 0:\n            nSpecies = 0\n        dbSpecList = birdDb.loc[ birdDb['Species'] == lSpecies].index\n        print(\"dbSpecList = \", dbSpecList)\n        if len(dbSpecList) > 1:           \n            nIndex = dbSpecList[nSpecies]\n        else:\n            nIndex = dbSpecList.index[0]        \n#                 \n    newBirdRow = birdDb.iloc[nIndex,:].copy()\n#     print(\"nIndex, Species, nSpec = \", nIndex , lSpecies , nSpecies )\n    return nIndex\ndef getBFinderRow( nIndex = -1, ecode = \"None\", nSongID = -1):\n    global bFinderDb\n    global newBFinderRow \n    if nIndex >= 0 and nIndex in bFinderDb.index: # if valid\n        ecode = bFinderDb.at[nIndex,'ecode']\n        nSongID = bFinderDb.at[nIndex,'nSongID']\n    else:\n        if nSongID < 0:\n            nSongID = 0\n        dbSpecList = bFinderDb.loc[ bFinderDb['ecode'] == ecode].index\n        if len(dbSpecList) > 1:           \n            nIndex = dbSpecList[nSongID]\n        else:\n            nIndex = dbSpecList.index[0]         \n    newBFinderRow = bFinderDb.iloc[nIndex,:].copy()\n#     print(\"nIndex, Species, nSpec = \", nIndex , ecode , nSongID )\n    return nIndex\n\ndef updateRow( nIndex = -1):\n    global birdDb\n    global newBirdRow\n    if nIndex < 0:\n        return -1\n    birdDb.iloc[nIndex,:] = newBirdRow\n    return 0\ndef updateBRow( nIndex = -1):\n    global bFinderDb\n    global newBFinderRow \n    if nIndex < 0:\n        return -1\n    bFinderDb.iloc[nIndex,:] = newBFinderRow\n    return 0    \n    \n\ndef predBird():\n    return\n\ndef runPipeline():\n    return\n\ndef trainBirdModel():\n    return\n\ndef enableAccell():\n    return\n\ndef splitWork():\n    return\n\ndef outputResults():\n    return","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"\n\ndef saveData(data, strFileName = \"proj_data\"):\n    filename = strFileName +\".csv\"\n    data.to_csv(filename, index=False)\n    return\n\ndef saveSubmission(sub):\n    saveData(sub, \"submission\")\n#     sub.to_csv('submission.csv', index=False)\n    return","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Stat Creation","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Create Full Path so we can access data more easily\n# ROOT_INPUTPATH = 'C:\\\\Users\\\\meckd\\\\PycharmProjects\\\\birdsong1\\\\data\\\\birdsong-recognition\\\\'\nROOT_INPUTPATH = '/kaggle/input/birdsong-recognition/'\n\nbase_dir = ROOT_INPUTPATH + 'train_audio/'\ntrain['full_path'] = base_dir + train['ebird_code'] + '/' + train['filename']\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def loadFile(file, sr=nSAMPLERATE):  # Add limit to length or file, or start/end..\n    x, x_sr = torchaudio.load(file)\n    x = x.numpy()\n\n    if x.ndim > 1:\n        x = np.mean(x, axis=0)\n    bReduceNoise = False\n    if bReduceNoise:\n    # Noise reduction\n        noisy_part = x[0:25000]  ### Warning need to id noisy part to pull\n        x = nr.reduce_noise(audio_clip=x, #Reduced noise here\n                        noise_clip=noisy_part, verbose=False)\n\n    if sr > 0 :\n        x = signal.resample(x, int(x.size*float(sr)/x_sr))\n        x_sr = sr\n        \n    return x, x_sr","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def freqCutOff(f,m): # trim low and too high freq data \n#     fCut = np.logical_and((30 < f), (f<10000)) # return array of t/f not quite right multiply parters\n    fCutLow = (30 < f) # return array of t/f\n    fCutHigh =  (f<10000) # return array of t/f  10k is about half of current 22k sampling\n    f = (f* fCutLow) * fCutHigh\n    m = (m* fCutLow) * fCutHigh\n    return f,m","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from scipy import fft, arange\ndef frequency_sepectrum(x, sf=nSAMPLERATE): \n    # https://stackoverflow.com/questions/53308674/audio-frequencies-in-python/53309191#53309191\n    \n    \"\"\"\n    Derive frequency spectrum of a signal from time domain\n    :param x: signal in the time domain\n    :param sf: sampling frequency\n    :returns frequencies and their content distribution\n    :NOTE: Hearing is generally 28Hz-20kHz, but freq cutoff with 22k sampling is 11kHz\n    \"\"\"\n    if len(x)<100:\n        return [],[]\n    \n    x = x - np.average(x)  # zero-centering\n\n    n = len(x)\n#     print(n)\n    k = arange(n)\n    tarr = n / float(sf)\n    frqarr = k / float(tarr)  # two sides frequency range\n\n    frqarr = frqarr[range(n // 2)]  # one side frequency range\n\n    x = fft(x) / n  # fft computing and normalization\n    x = x[range(n // 2)]\n    # Added by me to normalize:\n    x= abs(x)\n    x = x / (np.max(x)/2) # similar to 0-2 range\n\n    return freqCutOff(frqarr,x)  # return freq's and magnitudes after fCutting to limit range","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from scipy.signal import butter, lfilter, freqz\n# https://stackoverflow.com/questions/25191620/creating-lowpass-filter-in-scipy-understanding-methods-and-units\n\n\n## This code replaces above 4 functions\ndef butter_Xpass_filter(data, cutoff, fs, order=5, bHighPass=True): # cutoff is in Hz, fs is sampling rate, order is filter orderQQ  \n    nyq = 0.5 * fs\n    normal_cutoff = cutoff / nyq\n    if bHighPass==True:\n        btype='high'\n    else: # Low Pass Filter instead, (blocks high)\n        btype='low'\n    b, a = butter(order, normal_cutoff, btype=btype, analog=False)\n    y = lfilter(b, a, data)\n    return y","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"nFFTGraphs = 0\ndef graphFFT(freqs, mags): \n#     fig, ax = plt.subplots(1, figsize = (16, 9))\n#     fig.suptitle('Sound Waves', fontsize=16)\n    global nFFTGraphs\n    nFFTGraphs +=1\n    if (nFFTGraphs)>8:\n        return # limit graphing\n    fig2 = plt.figure(1, figsize = (12, 4))\n    plt.plot(freqs,mags)\n    plt.xlabel('frequency')\n    plt.ylabel(\"Magnitude\") \n    plt.show()\n    return","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def getMaxP(data):   # get max Power, indexP, and FreqP in this order...\n    mFreq, mPower, nMaxF = getMaxFrqPwr(data)\n    return mPower, nMaxF, mFreq        #fMaxP, nMaxPId, fMaxPFreq ## order change???","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def getMaxFrqPwr(data):  ## right now max power is always 2 due to scaling to 2\n    freqs, mags = frequency_sepectrum(data)\n    nMaxF =  (mags**2).argmax() # mags.argmax() #\n    mFreq = round(freqs[nMaxF],2)      #((freqs[nMaxF])*100//1)/100\n    \n    avgMag = np.mean(mags)*2 # trying to get more range\n    mPower = abs(50*(2-avgMag) )    ## mags[nMaxF] Always 2  # Diff mean to top times 100/2.0, so ranged to 100 max\n    \n    graphFFT(freqs, mags) # will graph until stopped by count\n    return mFreq, mPower, nMaxF","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def calcFreqStats(data,saveData = True, fMinTh=0.0, showGraph = False):\n    nLength = len(data)\n    if nLength<100:\n        return 0,0\n    \n    secData = nLength / nSAMPLERATE  \n    mFreq, mPower, nMaxF = getMaxFrqPwr(data)   \n#     print(\"sec long, maxP(of 100) = \", secData,  mPower)   \n    return mFreq, mPower","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# https://www.kaggle.com/seshganesh/dataprocessing\n\ndef calcFreqStats1(data, fMinTh=0.0, saveData = True, showGraph = False):\n    global lastData\n    if saveData == True:\n        lastData = data\n#     else:\n#         data = lastData # using saved data\n    lastData = data # making sure data gets here\n    if len(data)<100:\n        return 0,0\n        \n    fMinTh=0.0 # testing   \n    ##########################################################################\n    freqs, mags = frequency_sepectrum(data)\n\n    nMaxF = mags.argmax() # (mags**2).argmax() \n    mFreq = ((freqs[nMaxF])*100//1)/100\n    mPower = mags[nMaxF]\n    \n    return mFreq, mPower","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def addBFinderRow(nSongID=0, ecode='', file='', dbPrimData=[] , dbFreqs=[], dbFingerprint=[] ):\n    global bFinderDb\n#     global bFinder_columns\n\n    lenDBPRs = len(dbPrimData) \n    lenDBFs = len(dbFreqs)\n    lenDBFPs = len(dbFingerprint)\n    if lenDBPRs < 7:# Fill rows if needed\n        for row in range (7-lenDBPRs):\n            dbPrimData.append( 0.0 )\n    if lenDBFs < 10:        \n        for row in range (10-lenDBFs):\n            dbFreqs.append( 0.0 )\n    if lenDBFPs < 10:\n        for row in range (10-lenDBFPs):\n            dbFingerprint.append( 0.0 )\n\n    del dbFreqs[10:] # was behind and else...\n    del dbFingerprint[10:]\n    \n    bNewBFind_row = [nSongID, ecode,file]  # 0-2\n    bNewBFind_row.extend(dbPrimData)    # 3-9\n    bNewBFind_row.extend(dbFreqs)       # 10-19\n    bNewBFind_row.extend(dbFingerprint) # 20-29\n#     print(\"bNewBFind_row = \", bNewBFind_row)\n    thisBFindRow = pd.Series(data=bNewBFind_row, index=bFinder_columns)\n    \n    bFinderDb = bFinderDb.append(thisBFindRow, ignore_index=True)\n#     print(\"freqDb = \", bFinderDb)\n    \n    return","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def peakFingerprint(freqs, delays, mags):  # Note freqs is condensed freq, and delay is the delay before each Freq point\n    fp = []\n    fTotFSlope=0.0\n    fTotMSlope=0.0\n    fTotDelays = 0.0\n    frqN = len(freqs)\n    if frqN<1:\n        return fp\n    elif frqN == 1:\n        fp = [freqs[0],1,0] # Peak, slope, magn to next\n        \n    fplus1,dplus1,mplus1 = 0.0, 0.0, 0.0\n    for nFrq in range(frqN):\n        if frqN > nFrq+2:\n            fplus1 = freqs[nFrq+1]\n            dplus1 = delays[nFrq+1]\n            mplus1 = mags[nFrq+1]\n            \n            m = (fplus1 - freqs[nFrq])/(22000/(dplus1+1)) # added * (steps/22000) as run\n            fSlope = math.degrees(math.atan(m ))#//1 # angle from x axis\n            mMag = (mplus1 - mags[nFrq])/(22000/(mplus1+1)) # added * (steps/22000) as run\n            fMagSlope = math.degrees(math.atan(mMag ))#//1 # angle from x axis\n            \n            fTotFSlope += fSlope\n            fTotMSlope += fMagSlope\n            fTotDelays += delays[nFrq]\n\n    fp.append(fTotFSlope)\n    fp.append(fTotMSlope)\n    fp.append(fTotDelays)\n\n    return fp","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def getCrossingCount2(data): \n    nPoints = len(data)\n    maxPoint = abs(data).max()\n#      sum(zero_crossings)  returns boolean array \n    zero_crossings = sum(librosa.zero_crossings(data, pad=False))    #(x[nStart:nEnd], pad=False) # pad is if x[0] is counted as cross\n#     print(\"zero_crossings, nPoints = \", zero_crossings, nPoints)\n    fHz = (zero_crossings * nSAMPLERATE / (2* nPoints) ) # (2 * (len(chunk))))\n    return fHz, maxPoint","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# import numpy as np\n# from numpy import logical_and, average, diff\n\ndef freq_from_crossings(sig, fs=nSAMPLERATE):\n    \"\"\"Estimate frequency by counting zero crossings\n      \n    Doesn't work if there are multiple zero crossings per cycle.\n    \n    \"\"\"\n    indices = np.nonzero(np.diff(sig > 0))[0]\n#     indices = find(logical_and(sig[1:] >= 0, sig[:-1] < 0))\n    \n    # Linear interpolation to find truer zero crossings\n    crossings = [i - sig[i] / (sig[i+1] - sig[i]) for i in indices]\n    return fs / np.average(np.diff(crossings))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Going to try and standardize on 5 sec times, with overlap  NOTE 1/2 OVERLAP NOW ######\ndef calcFreqSweep(nDiv=100, data=[], chirp_data=[], bUseFFT=True, fMinTh=0.0): # divmod or // for int division\n    global lastData\n    freq = []\n    mag = []\n    nPoints = len(data)\n    if nPoints>100:\n        lastData = data\n    else:\n        nPoints = len(lastData)\n    \n    frMax,magMax = calcFreqStats(lastData, saveData=False)\n    magMin = fMinTh\n    \n    if nPoints > (nSAMPLERATE*5):\n        nPoints = (nSAMPLERATE*5) # 5 sec.\n        \n    nOverlap = 1     # 2 = 1/2 overlap, 1 for no overlap, etc\n    nStepSize = (nPoints // nDiv)*nOverlap\n#     print(\"nStepSize = \", nStepSize)\n    start = 0\n    nTotSteps = nDiv-nOverlap+1\n    for step in range(nTotSteps):\n        stop = start + nStepSize\n        if step == nDiv-1:\n            stop = nPoints\n            \n        #check if data over 0\n        m = np.max(chirp_data[start:stop])\n        if m >1:\n            \n            if bUseFFT == True:\n                f,_ = calcFreqStats(lastData[start:stop], fMinTh=fMinTh, saveData=True)\n            else:\n#                 f0, fSpacing = getCrossingCount(lastData[start:stop], fMinTh=0.0) \n                f, m2 = getCrossingCount2(lastData[start:stop]) \n#                 f = freq_from_crossings(lastData[start:stop]) \n#                 if m != m2:\n#                 print(\"Not Equal m, m2 = \",  m, m2)\n#                 print(\"Not Equal f0, f, f3 = \",  f0, \", \", f,\", \", f3)\n#                 m=0\n        else:\n            f = 0.0\n            \n#         print(\"calc Freq start, stop\", start, stop, f)\n        if (f <= 23.0) or (m2 < 0.25):## m2 added need to test\n            f = 0.0\n            m = 0.0\n        start = start + (nStepSize//nOverlap)\n        freq.append(f)\n        mag.append(m)\n#     if bUseFFT == True:\n#         print(\"FFT - freq found = \", freq) # output all Freq, delays\n#     else:\n#         print(\"Spacing - freq found = \", freq) # output all Freq, delays\n    return freqCompress(freq, mag)  #################### LOSING Magnitude 1-100 measurement here!!!!\n\n###################\n\n\ndef freqCompress(freq, mag):\n    delayDb = []\n    freqDb = []\n    magnDb = []\n    mag = mag\n    nDelayCount=0\n    nCount = 0\n    nFeqCount = 0\n    for (f,m) in zip(freq,mag):\n        if f>20:\n#             if nDelayCount>-1 or nCount==0:\n            delayDb.append(nDelayCount)\n            freqDb.append(f)\n            magnDb.append(m)\n            nDelayCount=0\n            nFeqCount+=1\n        else:\n            nDelayCount+=1\n        \n        \n        nCount +=1\n        if nFeqCount>=100:\n            break\n#     print(\"Freq, Delay = \", freqDb, delayDb) \n    if len(freqDb) < 3: ## hack for later indexing...\n        threeZeros = [0.0, 0.0, 0.0]\n        freqDb.extend(threeZeros)\n        magnDb.extend(threeZeros)\n        delayDb.extend(threeZeros)\n        \n    \n    fp = peakFingerprint(freqDb, delayDb, magnDb)\n    return freqDb, magnDb, fp\n##################\n\ndef freqCompress2(freq, mag): ## unused, experimental now\n    delayDb = []\n    freqDb = []\n    magnDb = []\n    mag = mag\n    nDelayCount=0\n    nCount = 0\n    nFeqCount = 0\n    lastF = -1\n    lastM = -1\n    magMax = max(mag)\n      \n    for (f,m) in zip(freq,mag):\n        if lastF < 0:\n            lastF = f\n            lastM = m\n        if (f>20) and (f>(magMax/4) ):\n            freqDb.append(f) # was lastF-f\n            magnDb.append(lastM-m)\n                       \n                       \n            riseM = (sum(magnDb))/(22000/(len(delayDb)*10 +1)) # added * (steps/22000) as run\n            fSlope = math.degrees(math.atan(riseM )) # angle from x axis  was //1\n            delayDb.append(fSlope)     ## This jacks up fingerprint      \n            nFeqCount+=1   \n            lastF = f\n            lastM = m\n#         else:                       \n        \n        nCount +=1\n        if nFeqCount>=100:\n            break\n            \n#     print(\"Freq, Delay = \", freqDb, delayDb) \n    if len(freqDb) < 3: ## hack for later indexing...\n        threeZeros = [0.0, 0.0, 0.0]\n        freqDb.extend(threeZeros)\n        magnDb.extend(threeZeros)\n        delayDb.extend(threeZeros)\n        \n    \n    fp = peakFingerprint(freqDb, delayDb, magnDb)\n    return freqDb, magnDb, fp #delayDb # fp ### HACKKKK  # F, D, Fp expected\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## New fingerprint\n    def fp(secWidth, secMiddle, totPts, midPts)\n*     Setup buckets for beginning, middle, end of peaks, with middle at top of main peak\n*     Buckets initially 20, 20 middle, 20 end\n*     Measure peak power and crossings (5 crossings is 2 waves), calc partial crossings too, with leftover percentage\n*     Be able to graph P, m for fp\n*     initial secWidth = 0.2, secMiddle = 0.04, so .005 sec bucket, .0025 sec/buck, .005\n \n>     This will be too wide for slow  and too narrow for high, but should be able to identify still...\n>     Potentially could vary based on avg Freq (by larger buckets...)","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"nCCRuns = 0\ndef getCrossingCount(data,fMinTh=25): \n    maxPoint = data.max()\n    nMaxPtIndex = data.argmax()\n    minThreshold = fMinTh \n    nPoints = len(data)\n    global nCCRuns\n    nCCRuns +=1\n    nStart = nMaxPtIndex -40\n    nEnd = nMaxPtIndex +40\n    if nStart < 0:\n        nStart = 0\n    if nEnd >= nPoints-1:\n        nEnd = nPoints-1\n        \n#     print(\"nSt, nEnd = \", nStart, nEnd)\n        \n    subData = data[nStart:nEnd]\n    nCrossings = 0\n    \n    bOver = subData > minThreshold\n    bLastValue = bOver[0]\n    nFirstID = -1\n    nLastID = -1\n    nIndex = 0\n    nBefore = 0\n    for bValue in bOver:\n        if bLastValue != bValue:\n            nCrossings += 1\n        bLastValue = bValue\n        if nFirstID < 0 and bValue >0:\n            nFirstID = nIndex # first up peak\n        if bValue > 0:\n            nLastID = nIndex # last up peak end\n        nIndex += 1\n        \n    if nCrossings >0:\n        fSpacing = (nEnd-nStart)/((nCrossings)/2)  # /2 to account for 4 crossings / lambda\n    else:\n        fSpacing = 1\n    \n    if fSpacing >0:\n        fHz = nSAMPLERATE/fSpacing\n    else:\n        fHz = 1\n    \n    fHz= (fHz*100//1)/100\n    fSpacing= (fSpacing*100//1)/100\n#     if nCCRuns < 5:\n#         print(\"Crossing Count peak data: \", subData)\n#         print(\"Crossings, fSpacing, st, end = \",nCrossings ,fSpacing, nFirstID, nLastID)\n    \n    return fHz, fSpacing","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def getPeakSpacing(data,fMinTh=0.0): \n    \n    maxPoint = data.max()\n    nMaxPtIndex = data.argmax()\n    minThreshold = fMinTh #maxPoint*(minThPercent/100)\n    fSpacing = 0.0\n    nPoints = len(data)\n    \n    nStart = nMaxPtIndex -200\n    nEnd = nMaxPtIndex +200\n    if nStart < 0:\n        nStart = 0\n    if nEnd > nPoints:\n        nEnd = nPoints\n        \n    subData = data[nStart:nEnd]\n#     print(\"subData = \", subData)\n#     print(\"minThreshold = \", minThreshold)   \n    \n    nUpCrossings, nDownCrossings = 0,0\n    nUpRows,nDownRows = 0,0\n    bUp,bDown,bUpCrossings,bDownCrossings = False,False,False,False\n\n    dbIndex = []\n    dbPower = []\n    dbSlope = []\n       \n    index = 0\n    nPeaks = 0\n    lastSlope = None\n    lastId = -1\n    lastPow = None\n    lastIdSlope = -1\n    for row in subData:\n        index +=1\n        if row > minThreshold:\n            if bUp == False:\n                bUp = True\n                bUpCrossing = True\n                nUpCrossings +=1\n#                 print(\"Index, Data = \", index, row)\n                dbIndex.append(index)\n                dbPower.append(row)\n                fSlope = row-lastPow\n                lastPow = row\n#             dbDiffPow.append( fSlope)\n                if nPeaks >0 and (dbPower[nPeaks] - dbPower[nPeaks-1])>1:\n                    dbSlope.append(dbIndex[nPeaks] - dbIndex[nPeaks-1])\n\n            else:\n                bUpCrossing = False\n            bDown = False\n            bDownCrossing = False\n            nUpRows +=1\n\n        if row < minThreshold:\n            if bDown == False and bUp == True:\n                bDown = True\n                bDownCrossing = True\n                nDownCrossings +=1\n            else:\n                bDownCrossing = False\n            bUp = False\n            bUpCrossing = False\n            nDownRows +=1\n            \n#     print(\"subData = \", subData)\n    nSubPoints = len(subData)\n    \n    fSpacing = nSubPoints/len(subData)\n   \n\n#     print(\"Number Up Rows, Number Down, nCrossings = \", nUpRows, \" , \", nDownRows,\" , \",nDownCrossings)\n\n    fDutyCycle = nUpRows / nDownRows\n\n    return fSpacing","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Calc Stats HERE --------------------","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def findFundFreq(fMaxPFreq,fFreqMean=0 ):\n    \n    ### Adding here now, will pull to function # range 1800-3600\n    nTypRange = -1\n    if fFreqMean >1650 and fFreqMean <1950:\n        nTypRange = 1\n    if fFreqMean >3300 and fFreqMean <3900:\n        nTypRange = 2\n            \n    if fMaxPFreq <= 225.0:  \n        fFundFreq = fMaxPFreq * 16.0\n        nFFmult = -4  # multiplier factor relative to FF range\n    elif fMaxPFreq <= 455.0:  \n        fFundFreq = fMaxPFreq * 8.0\n        nFFmult = -3\n    elif fMaxPFreq <= 900.0:  \n        fFundFreq = fMaxPFreq * 4.0\n        nFFmult = -2\n    elif fMaxPFreq > 1650 and fMaxPFreq <= 1800 and nTypRange == 1: # special case\n            fFundFreq = fMaxPFreq * 1.0\n            nFFmult = 0    \n    elif fMaxPFreq <= 1800.0: # 900 - 1799\n        fFundFreq = fMaxPFreq * 2.0\n        nFFmult = -1       \n    elif fMaxPFreq <= 3600.0:  # 1800 - 3600\n        fFundFreq = fMaxPFreq * 1.0\n        nFFmult = 0\n    elif fMaxPFreq > 3300 and fMaxPFreq < 3900 and nTypRange == 2: # special case\n            fFundFreq = fMaxPFreq * 1.0\n            nFFmult = 0  \n    elif fMaxPFreq <= 7200.0:  # 3600 ++\n        fFundFreq = fMaxPFreq * 0.5\n        nFFmult = 1\n    else:  \n        fFundFreq = fMaxPFreq * 0.5  ## Most likely better not adding another level\n        nFFmult = 1    \n    print(\"ff = \", fFundFreq, nFFmult)\n    return fFundFreq, nFFmult\n           ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#  calcStats(audio_bird, filename=file, ecode = ecode, nIndex=objSong.nIndex)\n##  ### This is fed 5 sec sections of data, file is only for reference\ndef calcStats(data, audio_chirp, minThPercent=25, filename=\"\", ecode = \"\", nIndex=0 , fDutyCycle = \"\", nIter=1):\n    global bPREDICTRUN\n\n    if len(filename)>5:\n        file = os.path.basename(filename) # this is only for db name, file already read\n    else:\n        file = \"NA\"\n#     fDutyCycle = params\n    \n    nLength = len(data)\n    if nLength > (nSAMPLERATE*1):\n        nEndLength = nLength - int(nSAMPLERATE*0.5) # 5 sec.\n    else:\n        nEndLength = nLength\n        \n    nMaxAudioP = audio_chirp[:nEndLength].argmax()\n\n    # Now set to 1/2 sec plus and minus from peak\n    if nMaxAudioP > int(nSAMPLERATE*0.5):\n        nMidStart = nMaxAudioP -int(nSAMPLERATE*0.5) # set back by 1/2 sec\n    else:\n        nMidStart = 0\n        \n    nMidEnd = nMidStart+int(nSAMPLERATE*0.5)\n    \n    fMaxP, nMaxP, fMaxPFreq = getMaxP(data[nMidStart:nMidEnd])\n    \n    dbF, dbD, dbFp = calcFreqSweep(40, data=data[nMidStart:nMidEnd], chirp_data=audio_chirp[nMidStart:nMidEnd], bUseFFT=False, fMinTh=0.3*fMaxP) # this is file based\n    #######################################\n    # dbPrimData = [ 'primFr', 'avgFr','primSl','dCycle', 'nSegm','fFundFreq','nFFmult', ]\\\n    fFreqMean = 0.0\n    nFrq = 0\n    for item in dbF:\n        if item>30 and item < 10000:\n            fFreqMean += item\n            nFrq += 1\n    if nFrq>0:\n        fFreqMean = fFreqMean/nFrq  # was np.nanmean(dbF)\n#         print(\"mean, n  = \", fFreqMean, nFrq)\n    percDiff =  (abs(fMaxPFreq-fFreqMean)/fFreqMean)  ## could if 1.0 then /2, (.5 to x2) etc....\n    \n    if (fFreqMean>800):  ### Testing catching second harmonics\n        if fMaxPFreq > fFreqMean and percDiff > 0.95 and percDiff < 1.05 :\n            fMaxPFreq = fMaxPFreq/2 ## suposing 2nd harmonic\n            print(\"fMaxPFreq = fMaxPFreq/2 ## suposing reverse 2nd harmonic = \",fMaxPFreq,\" , \", fFreqMean )\n        if fMaxPFreq < fFreqMean and percDiff > 0.46 and percDiff < 0.54 :\n            fMaxPFreq = fMaxPFreq*2 ## suposing reverse 2nd harmonic\n            print(\"fMaxPFreq = fMaxPFreq*2 ## suposing reverse 2nd harmonic = \",fMaxPFreq,\" , \", fFreqMean )\n            \n    fFundFreq,nFFmult = findFundFreq(fMaxPFreq,fFreqMean )\n    \n#     ### Adding here now, will pull to function # range 1800-3600\n#     nTypRange = -1\n#     if fFreqMean >1650 and fFreqMean <1950:\n#         nTypRange = 1\n#     if fFreqMean >3300 and fFreqMean <3900:\n#         nTypRange = 2\n            \n#     if fMaxPFreq <= 225.0:  \n#         fFundFreq = fMaxPFreq * 16.0\n#         nFFmult = -4  # multiplier factor relative to FF range\n#     elif fMaxPFreq <= 455.0:  \n#         fFundFreq = fMaxPFreq * 8.0\n#         nFFmult = -3\n#     elif fMaxPFreq <= 900.0:  \n#         fFundFreq = fMaxPFreq * 4.0\n#         nFFmult = -2\n#     elif fMaxPFreq > 1650 and fMaxPFreq <= 1800 and nTypRange == 1: # special case\n#             fFundFreq = fMaxPFreq * 1.0\n#             nFFmult = 0    \n#     elif fMaxPFreq <= 1800.0: # 900 - 1799\n#         fFundFreq = fMaxPFreq * 2.0\n#         nFFmult = -1       \n#     elif fMaxPFreq <= 3600.0:  # 1800 - 3600\n#         fFundFreq = fMaxPFreq * 1.0\n#         nFFmult = 0\n#     elif fMaxPFreq > 3300 and fMaxPFreq < 3900 and nTypRange == 2: # special case\n#             fFundFreq = fMaxPFreq * 1.0\n#             nFFmult = 0  \n#     elif fMaxPFreq <= 7200.0:  # 3600 ++\n#         fFundFreq = fMaxPFreq * 0.5\n#         nFFmult = 1\n#     else:  \n#         fFundFreq = fMaxPFreq * 0.5  ## Most likely better not adding another level\n#         nFFmult = 1    \n            \n    dbPrimData = [fMaxPFreq, fFreqMean, fMaxP,fDutyCycle, nIter, fFundFreq, nFFmult] # end rough slope\n\n    if fMaxP >= 70.0 and fMaxPFreq >= 100.0 and fFreqMean >= 100.0:  ### DETERMINE IF ADDED TO DB !!!!!\n#     if (bPREDICTRUN == True) or (abs(fMaxPFreq-fFreqMean)/fFreqMean) < 0.4 and fMaxP >= 90.0 :  ### DETERMINE IF ADDED TO DB !!!!!\n        nECode = enumECodes[ecode] \n        fCode_Fn = ecode + '_' + os.path.splitext(file)[0]\n        addBFinderRow(nSongID=nIndex, ecode=nECode, file=fCode_Fn, dbPrimData=dbPrimData , dbFreqs=dbF, dbFingerprint=dbFp )\n    elif (bPREDICTRUN == True):\n        nECode = enumECodes[ecode] \n        fCode_Fn = ecode + '_' + os.path.splitext(file)[0]\n        dbPrimData = [0.0, 0.0, 0.0, 0.0, nIter, 0.0, 0] # force to \"nocall\" hopefully...\n        addBFinderRow(nSongID=nIndex, ecode=nECode, file=fCode_Fn, dbPrimData=dbPrimData , dbFreqs=dbF, dbFingerprint=dbFp )\n    elif fMaxP < 60.0 or fMaxPFreq < 100.0 or fFreqMean < 100.0:\n        nECode = enumECodes[\"nocall\"] # set as nocall\n        fCode_Fn = ecode + '_' + os.path.splitext(file)[0]\n        print(\"------- NO CALL ----- = Freq, avFre, fMaxP\" ,fMaxPFreq,\", \", fFreqMean,\" - \", fMaxP )\n        addBFinderRow(nSongID=nIndex, ecode=nECode, file=fCode_Fn, dbPrimData=dbPrimData , dbFreqs=dbF, dbFingerprint=dbFp )\n    else:  ## could catch low, say < 60 power level and label as \"nocall\" ********************************\n        print(\"@@@@@@@ Failing save Criteria = Freq, avFre, fMaxP\" ,fMaxPFreq,\", \", fFreqMean,\" - \", fMaxP ) # Need to deal with over dropping\n\n#     print(\"db Diff Spacing = \", fSpacing)\n\n    newRow()\n    addRowValue('ecode', ecode)\n    addRowValue('file', file)\n    addRowValue('sPitch', fMaxP )\n    addRowValue('sSpacing', fFreqMean )  \n    addRowValue('sDutyCycle', fMaxPFreq )\n    nRowID = appendRow()\n        \n    return nRowID\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def showBirdSong(audio_file, sr_file, name = 'bird_file'):\n    \n    fig, ax = plt.subplots(1, figsize = (12, 4)) # 16,9 works\n    fig.suptitle('Sound Waves', fontsize=16)\n\n    librosa.display.waveplot(y = audio_file, sr = sr_file, color = \"#A300F9\", ax=ax);\n#     librosa.display.waveplot(y = audio_cangoo, sr = sr_cangoo, color = \"#4300FF\", ax=ax[1])\n#     librosa.display.waveplot(y = audio_haiwoo, sr = sr_haiwoo, color = \"#009DFF\", ax=ax[2])\n#     librosa.display.waveplot(y = audio_pingro, sr = sr_pingro, color = \"#00FFB0\", ax=ax[3])\n#     librosa.display.waveplot(y = audio_vesspa, sr = sr_vesspa, color = \"#D9FF00\", ax=ax[4]);\n\n    ax.set_ylabel(name, fontsize=13)\n    plt.show()\n    return","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def audioRanging(data, nPercCutoff=20): # Returns mirror of array with range 0-\n    fMax = np.max(data)\n    data_range = ((data.copy())*(100/fMax))//1\n    data_range_abs = abs(data_range)\n\n    chirp_filter = (data_range_abs >= nPercCutoff) # 0 and 1\n    chirp_data = chirp_filter * data_range # data below Cutoff set to 0\n    fDutyCycle = round( (sum(chirp_filter) / len(chirp_filter))*100, 2 )\n#     fDutyCycle = ((sum(chirp_filter) / len(chirp_filter))*10000//1)/100  #round(fSecFile,2)\n    \n#     if nPipeRuns < 5:\n#         print(\" sum(chirp_filter), len(chirp_filter), fDutyCycle  \", sum(chirp_filter), len(chirp_filter), fDutyCycle)\n        \n    return chirp_data, fDutyCycle, data_range  ## chirp is 0-2 with zero where under cutoff, range is 0-2","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Steps for 5 second pass of just data:\n\n1. Read whole file, store in global? - readBirdFile returns array of 5 sec clips\n2. Scan file for quiet/loud spots with filter, filter to zero (may clip part of valid peaks near zero, think waves) - done in above\n3. For a given 5 seconds try to find peak frequency in first 3 seconds\n4. Center just before start of peak and go to end of file\n5. Record fingerprint of peak+ 2 sec or so (allow for later peaks too, posibly by using rest of file)\n6. Hmm, see if I can lock in some portion of the song birds with known signatures, I could then scan for only those and label rest as no song.\n7. Match top 5 birds to a given 5 sec. segment, use threshold to ignore all, or keep some\n8. For site 3 a post process will be needed to add all birds to the site as a whole (as whole file has single bird list)\n9. For sites 1,2 the birds will need to be chosen and affixed to the right rows.  (I could find bird at a given timestamp possibly)","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def load_test_clip(path, start_time, duration=5):\n    return librosa.load(path, offset=start_time, duration=duration, sr=nSAMPLERATE,mono=True, res_type='kaiser_fast') # [0] Kaiser_fast resampling is faster\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"\nlastFile = \"\" \n_y_bird = []\n_sr_bird = 0\ndef readBirdFile(file, secIteration=5.0, bIterate = True, secStart=0.0, secDur=None): # Returns array of 5 sec sections of file\n    if secStart == None:\n        secStart = 0.0\n    secEnd = secIteration # updated later with end time if less than secIter..\n    global _y_bird\n    global _sr_bird\n    global lastFile\n    bKeepBirds = False\n    bKeepFileData = len(lastFile)>5 and (lastFile.lower() == file.lower())   \n#     bKeepFileData = False\n    \n    # Importing the file\n#     if 1:\n    try:\n        if (bKeepFileData == False) or (len(_y_bird)<100): # if true use existing data     \n            if (bPREDICTRUN == True):       \n                _y_bird, _sr_bird = load_test_clip(file, start_time=secStart, duration=None)\n            else:\n                _y_bird, _sr_bird = load_test_clip(file, start_time=secStart, duration=None)\n        else:\n            print('Keeping lastData...lastFile, file',lastFile, file )\n            print(\"From Que st, dur = \",secStart,secDur )\n            print(\"_y_bird = \", _y_bird[:5])\n            \n        lastFile = file\n        if (secDur != None):\n            y_bird_loc = _y_bird[int(secStart*nSAMPLERATE):int((secStart+secDur)*nSAMPLERATE)] # local segment\n        else:\n            y_bird_loc = _y_bird[int(secStart*nSAMPLERATE):] \n            \n        chirp_data,fDutyCycle,_ = audioRanging(y_bird_loc) ## filter to int 1-100 for over threshold data\n        y_bird = chirp_data\n        if bFILTER == True: #\n            y_bird = butter_Xpass_filter(y_bird, 900, nSAMPLERATE, order=5)############ LOWPASS FILTER START level\n#             y_bird = butter_lowpass_filter(y_bird, 900, nSAMPLERATE, order=5)############ LOWPASS FILTER START level            \n            fMaxP, nMaxP, fMaxPFreq = getMaxP(y_bird)\n            print(\"At ReadBirdFile finished butter filter...fr = \", fMaxPFreq)\n            if fMaxPFreq <1100:\n                print(\"## RERUN at 180 - First Freq = \", fMaxPFreq)\n                y_bird = butter_Xpass_filter(y_bird, 180, nSAMPLERATE, order=5)############ LOWPASS FILTER START level\n                fMaxP, nMaxP, fMaxPFreq = getMaxP(y_bird)\n                \n                if fMaxPFreq <500:\n                    print(\"Second Freq = \", fMaxPFreq)\n                    y_bird = butter_Xpass_filter(y_bird, 30, nSAMPLERATE, order=5)############ LOWPASS FILTER START level\n                    fMaxP, nMaxP, fMaxPFreq = getMaxP(y_bird)\n                    print(\"Third Freq = \", fMaxPFreq)\n                               \n        nPoints = len(y_bird)\n        fSecFile = nPoints/nSAMPLERATE\n#         print(\"WHOLE #### File max P values p,n,f , TotSecs= \",fMaxP, nMaxP, fMaxPFreq, round(fSecFile,2) )\n\n        nTotIter = math.ceil(fSecFile / secIteration)\n        if bPREDICTRUN == False:\n            if bIterate == True:\n                secLastIter = fSecFile % secIteration  \n                if secLastIter < 3.0 and fSecFile > secIteration:\n                    nTotIter = nTotIter-1 # Last iter too short, let's skip it  ### Keep if Submission file??\n            else:\n                nTotIter = 1\n            if nTotIter > nMAXSONGITER:\n                nTotIter = nMAXSONGITER # limit to this number of segments\n        else:  # pred run\n            bKeepBirds = nTotIter > 1 # site 3 likely\n            \n            \n            \n        y_bird_arr = np.empty((int(nTotIter), math.ceil(secIteration*nSAMPLERATE)-1),dtype=float)\n        y_chirp_arr = y_bird_arr.copy()\n        for nIter in range(nTotIter):\n#             print(\"Read file, parsing - #\", nIter, \" - of \", nTotIter)\n            if fSecFile <= secEnd:\n                secEnd = fSecFile\n\n#             print(\"sec Start, End = \",secStart , secEnd )\n            audio_bird =  (y_bird[int(secStart*nSAMPLERATE):int(secEnd*nSAMPLERATE)-1]).copy() # 5 sec song\n            audio_chirp =  (chirp_data[int(secStart*nSAMPLERATE):int(secEnd*nSAMPLERATE)-1]).copy() # 5 sec w/zeros\n            if audio_bird.shape==(secIteration*nSAMPLERATE-1,):\n                y_bird_arr[nIter]= audio_bird.copy()   ## Building Array of 5 second segments\n                y_chirp_arr[nIter]= audio_chirp.copy()\n            else:\n                nShort = int((secIteration*nSAMPLERATE-1)- len(audio_bird))\n                audio_bird = np.append(audio_bird, np.zeros((nShort)), axis=None)\n                audio_chirp = np.append(audio_chirp, np.zeros((nShort)), axis=None)\n#                 print(\"Finally Else case .....shape = \", audio_bird.shape)\n                if audio_bird.shape==(secIteration*nSAMPLERATE-1,):\n                    y_bird_arr[nIter]= audio_bird.copy()\n                    y_chirp_arr[nIter]= audio_chirp.copy()\n#                     print(\"Odd shape - success!\")\n            secStart = secEnd\n            secEnd = secEnd + secIteration\n    except Exception as e:\n        print(\"###### ERROR READING - file - \", file)\n        print(\"EXCEPTION  = \",e)\n        return [], [], -1, False\n#     except:\n#         print(\"###### ERROR READING - file - \", file)\n# #         bRUNNING = False\n        \n#     print(\"y_bird_arr = \", y_bird_arr)\n\n    return y_bird_arr, y_chirp_arr, fDutyCycle, bKeepBirds","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"\nnPipeRuns = 0\ndef birdPipeline( objSong=newToProcessRow ):   #file, idTrain = \"\"):\n    global nPipeRuns\n    global audio_bird_arr\n    global audio_chirp_arr\n    global fDutyCycle\n    nPipeRuns +=1\n#     global train\n    global bPREDICTRUN\n#     print(\"Obj = \", objSong)\n#     print(\"train = \" , train[:3])\n\n    file = objSong.file\n    idTrain = objSong.FileRefId\n    secStart = objSong.startSec\n    secDur = objSong.duration\n    nIter = objSong.nIter\n    nLoc = objSong.nLoc\n\n    if bPREDICTRUN == False:\n        ecode = train.iloc[objSong['nIndex'],:]['ebird_code'] # keeps not changing ### ERROR HERE on Submit code\n        print(\"Bird Pipeline --- eCode  = \", ecode,\" -  pipe run #\", nPipeRuns,\"\" )\n    else:\n        ecode = 'unknown'  #enumECodes['nocall'] #262 #test.iloc[objSong['nIndex'],:]['ebird_code']\n        print(\"Bird Pipeline --- eCode  = Unk, file = \",file, \" -  pipe run #\", nPipeRuns,\"\" )\n    #print(\"string ecode = \", ecode)\n\n   \n#     if nIter < 1:\n    # Importing the file\n    audio_bird_arr,audio_chirp_arr,fDutyCycle, bKeepBirds = readBirdFile(file, secIteration=5.0, bIterate = True, secStart=secStart, secDur=secDur) ##### READ FILE LOOP\n    if fDutyCycle < 0:\n        return nPipeRuns # ERROR on read try\n    if bKeepBirds:\n        print(\"KEEP BIRDS set #############################\")\n        print(\"From Que st, dur = \",secStart,secDur )\n    \n    nIters = audio_bird_arr.shape[0]\n#     print(\"number of iterations = \", nIters)\n    audio_bird = audio_bird_arr[0]\n    audio_chirp = audio_chirp_arr[0]\n      \n    bShow = not bSUBMITRUN #bFASTRUN    \n    for nIter in range(nIters):  \n        audio_bird = audio_bird_arr[nIter]\n        audio_chirp = audio_chirp_arr[nIter]\n        if bShow == True and nPipeRuns<5:\n            showBirdSong(audio_bird, nSAMPLERATE, name = idTrain)\n#         print(\"found ecode = \", ecode,\" - loopSongID = \", objSong.nIndex)\n        ### Below is fed 5 sec sections of data, filename passed is only for reference\n        calcStats(audio_bird, audio_chirp, filename=file, ecode = ecode, nIndex = objSong.nIndex,fDutyCycle=fDutyCycle, nIter=nIter )  \n    \n    \n    return nPipeRuns\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#CPU count\nnCPUs = cpu_count()\nprint(\"cpu_count = \", nCPUs)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"nTotFiles = 0\n\ndef addBirdFileToQ(file, idTrain = \"\",fStartSec = 0.0,fDur = None, nIter=0, nLoc=1): #,'startSec', 'duration' \n    global qToProcess\n    global nTotFiles\n    \n    if len(file)>5:\n        file_short = os.path.basename(file)\n    else:\n        file_short = \"NA\"\n        return -1\n    nTotFiles +=1\n    print(\"Added to q st, dur = \",fStartSec,fDur )  \n    newToProcessRow = pd.Series(data=qToProcess_row, index=qToProcess_columns) #, 'nIter', 'nLoc'\n    newToProcessRow['nIndex'] = idTrain\n    newToProcessRow['file'] = file\n    newToProcessRow['FileRefId'] = str(file_short) +\"_\" + str(idTrain)\n    newToProcessRow['nStatus'] = 0\n    newToProcessRow['startSec'] = fStartSec\n    newToProcessRow['duration'] = fDur\n    newToProcessRow['nIter'] = int(nIter)\n    newToProcessRow['nLoc'] = int(nLoc)\n    qToProcess.put(newToProcessRow)\n    qSize = qToProcess.qsize()\n\n    print(\"To Process Que size = \", qSize)\n    return nTotFiles","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"\n\ndef workerFileQ():\n    items = []\n    bTask = False\n    global bRUNNING\n    bRUNNING = True\n    print(\"Worker Thread Starting **********\")\n    while bRUNNING == True:\n        bTask = False\n\n        lSong = qToProcess.get()\n        bTask = True\n          \n        nSongs = birdPipeline(lSong)  #.file, lSong.FileRefId)\n        qToProcess.task_done()\n#         print(\"Number = \" + str(nSongs) + \", - Finished \" , lSong.FileRefId,\" \")\n        if (qToProcess.qsize() < 1):\n            time.sleep(1000/1000)\n        \n        if (qToProcess.qsize() < 1):\n#             print(\"Q is done I believe..\")\n            time.sleep(3000/1000)\n            if (qToProcess.qsize() < 1):\n                print(\"Task thread timeout to finish....\")\n                break\n    print(\"Worker Thread Closing.... XXXXXXXXXXX\")\n        ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def addToSongQue(file, ecode='None'):\n    newSongsRow = pd.Series(data=queSongs_row, index=queSongs_columns)\n    newSongsRow['file'] = file\n    newSongsRow['ecode'] = ecode\n    return","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# MAIN BIRD LOOP \n# =======================","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# dbEbirdCodes[120]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# loopSongIDs = train[train['ebird_code'] == dbEbirdCodes[186]][73:101].index\n# for loopSongID in loopSongIDs:\n#     print(\"song ids = \",loopSongID)\n    \n# print(\"Done\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"##### code = \"\"\ndef birdLoop():\n    \n    startTime = time.time()\n    avgTime = 0.0\n    \n    bSpecialList = False\n    bSList = [67,136]\n    #### FOR ALL run, index,nMax, perCode => 0, 999, variable by slice\n    nStartCodeIndex = 15 # lotduc 147 i believe # Full missing: 76,82, 96, 120\n    nMaxCodes = 2     # of Species tested\n    nMaxPerCode = 1    # song files per Species\n    nMaxPerHardCode = nMaxPerCode*3\n    global dbEbirdCodes\n    global bRUNPARALLEL\n    global nCPUs\n    initRun()\n    \n    bRUNPARALLEL = False\n    global code\n    global loopSongID\n    loopSongID = 0\n    print(\"Init done, starting filling que....\")\n\n    if True:\n        nLoops = 0\n        nRunSongs = 0\n        for code in dbEbirdCodes:\n            nEcode = enumECodes[code]\n#             if (bSpecialList == True) and (code in bSList) # if item in my_list:\n            if ((bSpecialList == True) and (nEcode in bSList)) or (nLoops >= nStartCodeIndex and nLoops < nMaxCodes+nStartCodeIndex):\n                \n                if nEcode in [67, 76, 77, 82, 96, 120, 136, 152]: # 67, 136 missing from full set\n                    nMaxPer = nMaxPerCode  ###### nMaxPerHardCode\n                    print(\"Doing harder code:....\", code)\n                else:\n                    nMaxPer = nMaxPerCode\n#                     print(\"NOT - extra codes....\", code)\n                bAllSongs = False\n                if bAllSongs == True:\n                    loopSongIDs = train[train['ebird_code'] == code][50:nMaxPerCode].index #### MODIFY here for partial sets -50:nMax...\n#                     loopSongIDs = train[train['ebird_code'] == code][25:].index\n                else:\n                    loopSongIDs = train[train['ebird_code'] == code].sample(nMaxPer, replace=True).index\n\n                for loopSongID in loopSongIDs:\n                    bQrows = True\n                    if bQrows == True:\n                        rowSong = train.iloc[int(loopSongID) ]\n                        loopSong = rowSong['full_path']\n                        fDur = rowSong['duration']\n#                     Qrows, nQrows = createQrow(train, int(loopSongID) ) ## added nIter, seconds, path\n                        Qrows, nQrows = trainCreateQrow(loopSong, duration=fDur)\n                        for nRowId in range(len(Qrows)):\n                            fStartSec = int(Qrows.loc[nRowId,['seconds']]) - 5.0 #0.0 ## not specified in test\n                            fDur = 5.0\n                            nIter = int(Qrows.loc[nRowId,['nIter']]) \n#                             print(\"loopSongID: fSsec, nIter = \",loopSongID,\": \",fStartSec, nIter )\n                            addBirdFileToQ(loopSong, idTrain=loopSongID,fStartSec = fStartSec,fDur = fDur,nIter=nIter)                            \n                    \n                    else:\n                        rowSong = train.iloc[int(loopSongID) ]\n                        loopSong = rowSong['full_path']\n                        fStartSec = 0.0 ## not specified in test\n                        fDur = rowSong['duration']\n    \n                        addBirdFileToQ(loopSong, idTrain=loopSongID,fStartSec = fStartSec,fDur = fDur)\n                    \n                    nRunSongs += 1\n                    totTime = (time.time() - startTime)\n                    avgTime = totTime/nRunSongs\n#                     print(\"Average, Total Loop Time = \", avgTime,\" , \", totTime)\n            nLoops += 1  \n   \n            \n    # block until all tasks are done\n    qToProcess.join()\n    print('All work completed')\n    bRUNNING = False\n\n    endTime = time.time()\n    print(\"Total time = {:.4f}\".format(endTime-startTime))\n    print(\"Total time = \", (endTime-startTime))\n\n    return","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"bad file lotduc/XC195038.mp3","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"bPREDICTRUN = False\nbSkipBLoop = True\nif (bPREDICTRUN == False) and (bSkipBLoop == False):\n    birdLoop() # Skip usually\n    saveData(bFinderDb, strFileName = \"Bproj_data\")\n\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"birdDb.head(2)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Results close, but only good for first row of file currently I believe, but getting close.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# birdDb.tail(20)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"bFinderDb.head(30)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"bFinderDb.tail(30)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Model NN","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## StratifiedKfold","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"https://towardsdatascience.com/writing-your-first-neural-net-in-less-than-30-lines-of-code-with-keras-18e160a35502\n\nThis might be easier....","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"from keras import models\nfrom keras import layers\nfrom keras.utils import to_categorical\n\nnTrainCol = 27\n# (train_images, train_labels), (test_images, test_labels) = mnist.load_data()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"bUseDisk = True ## using last data set vs new for now\nif bUseDisk == True:\n#     train_all = train_part1.append(train_part2)\n#     train_all = train_all.append(train_part3)\n#     train_all = train_all.append(train_part4)\n#     train_all = train_all.append(train_part5)\n#     train_all = train_all.append(train_part6)\n#     train_all = train_all.append(train_part7)\n#     train_all = train_all.append(train_part8)\n#     train_all = train_all.append(train_part9)\n    \n    train_all = train_part1b.append(train_part2b)\n    train_all = train_all.append(train_part3b)\n    train_all = train_all.append(train_part4b)\n    train_all = train_all.append(train_part5b)\n    train_all = train_all.append(train_part6b)\n    train_all = train_all.append(train_part7b)\n#     train_all = train_all.append(train_part8)\n#     train_all = train_all.append(train_part9)\n    # train_part5\nelse:\n    train_all = bFinderDb.copy()\n\ntrain_all.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# train_all.loc[(train_all['ecode'] ==  enumECodes['amebit'])  ]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"countTrainBirds(train_all)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"Shape Before Filter = \", train_all.shape)\ntrain_all_filtered = train_all.loc[(train_all['ecode'] <  enumECodes['nocall']) & (train_all['primSl'] >  50.00) ]# was != working\nnECodes = len(train_all_filtered['ecode'].unique())\n# if nECodes < nEbirdCodes-2:\n#     print(\"1 - too few birds x vs y\",nECodes, nEbirdCodes-1 )\n#     train_all_filtered = train_all.loc[(train_all['ecode'] !=  enumECodes['nocall']) & (train_all['primSl'] >  70) ] # 87.83- Works\n# nECodes = len(train_all_filtered['ecode'].unique())\n# if nECodes < nEbirdCodes-2:\n#     print(\"2 - too few birds x vs y\",nECodes, nEbirdCodes-1 )\n#     train_all_filtered = train_all.loc[(train_all['ecode'] !=  enumECodes['nocall']) & (train_all['primSl'] >  65) ]   \nnECodes = len(train_all_filtered['ecode'].unique())\nprint(\"====Compare birds x vs y\",nECodes, nEbirdCodes-1 )\n# Note above use & opperator inside loc for compare vs 'and' in normal python\n# train_all_filtered = train_all\nprint(\"Shape after nocall removed = \", train_all_filtered.shape)\ntrain_all_filtered.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# train_all_filtered.sort_values(['ecode'], ascending=False, inplace=True)\n# train_all_filtered2 = train_all_filtered.copy()\n# dfNUnique = train_all_filtered2['ecode'].value_counts()\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# print(dfNUnique[-10:])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"## Filter out all but best 20 songs\ntrain_all_filtered.sort_values(['primSl'], ascending=False, inplace=True) # .sort_values(['primSl'], ascending=False, inplace=True)\ntrain_all_filtered2 = train_all_filtered.copy()\ndbTrainECodes = train_all_filtered2['ecode'].unique()\nprint(\"Train Ecodes unique = \", len(dbTrainECodes))\ndrop_indexes = []\nnDropThreshold = 100\nfor nCode in dbTrainECodes:\n    ecode_best = train_all_filtered2.loc[(train_all_filtered2['ecode'] == nCode )].index\n#     bestI = (train_all_filtered2[ecode_best]).sort_values(['primSl'], ascending=False).index\n#     ecode_best.sort_values(['ecode'], ascending=False, inplace=True)\n    \n#     data.sort_values(\"Name\", axis = 0, ascending = True, \n#                  inplace = True, na_position ='last') \n\n#     .sort_values('ecode', ascending=False, inplace=True)\n#     if ecode_best != None:\n    nLen = len(ecode_best)\n#     print(\"code = \", enumECodesRev[nCode],\" - , actual = \",nLen )\n#     else:\n#         nLen = 0\n#         print(\"None at - \", nCode,ecode_best )\n#     DataFrame.sort_values(by, axis=0, ascending=True, inplace=False, kind='quicksort', na_position='last')\n    if nLen > nDropThreshold:\n        ecode_drop = ecode_best[:nDropThreshold]\n        drop_indexes.extend(ecode_drop)\n#         train_all_filtered2.drop(ecode_drop, inplace=True) ## comment/uncomment to DROP ### NOT WORKING CORRECT DROPPING WRONG ITEMS\n    else:\n        print(\"code = \", enumECodesRev[nCode],\" - short of \",nDropThreshold,\", actual = \",nLen )\n#     ecode_best = train_all_filtered2.loc[(train_all_filtered2['ecode'] == nCode )].index\n#     nLen = len(ecode_best)\n#     print(\"AFTER code = \", enumECodesRev[nCode],\" -  of 20, actual = \",nLen ) \nprint(\"Shape after below \",nDropThreshold,\" removal (of \",train_all_filtered.shape[0],\")= \", train_all_filtered2.shape) \n\n\n# print(\"Length of drops = \", len(drop_indexes))\n# train_all_filtered2 = train_all_filtered2.iloc[drop_indexes]\n# # train_all_filtered2.drop(drop_indexes, inplace=True, )   \n# print(\"Shape after below 20 removal (of 58413)= \", train_all_filtered2.shape)   \n# len(train_all_filtered2['ecode'].unique() ) \n    ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_all_filtered2.head(30)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#  train , test is our sets here\n\n# trainData = train_all_filtered2.loc[:,'primFr':].fillna(0)\n# trainData = train_all_filtered2.loc[:,['avgFr','dCycle','primSl']].fillna(0) ## Limited columns \ntrainData = train_all_filtered2.loc[:,['fFundFreq','dCycle','primSl']].fillna(0) ## Limited columns avgFr ## 20% plus!! at 15 sample rate\n\n# trainData = trainData['primSl','dCycle','fFundFreq','nFFmult']\n# train = train_full1.loc[:,'primFr':'dCycle'] \ntrainData_labels = train_all_filtered2['ecode'].fillna(0)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"trainData.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Model includes - may move up ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# https://scikit-learn.org/stable/auto_examples/classification/plot_classifier_comparison.html#sphx-glr-auto-examples-classification-plot-classifier-comparison-py\nfrom sklearn.linear_model import LogisticRegression\nfrom sklearn.discriminant_analysis import LinearDiscriminantAnalysis\nfrom sklearn.neighbors import KNeighborsClassifier\nfrom sklearn.tree import DecisionTreeClassifier\nfrom sklearn.naive_bayes import GaussianNB\nfrom sklearn.svm import SVC\nfrom sklearn.ensemble import GradientBoostingClassifier # XGB trying to add\n\nfrom sklearn.preprocessing import StandardScaler\nfrom sklearn import model_selection\n#  # from Titanic example https://www.kaggle.com/kaerunantoka/titanic-pytorch-nn-tutorial\n\nfrom sklearn.preprocessing import LabelEncoder\n\n# from sklearn.model_selection import train_test_split\nfrom sklearn.metrics import classification_report\nfrom sklearn.metrics import confusion_matrix\nfrom sklearn.metrics import accuracy_score\n\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Scaling","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"numeric_cols = list(set(trainData.columns) )\n# train[numeric_cols] = train[numeric_cols].fillna(0)\nprint('scaling numeric columns')\n\nscaler = StandardScaler()\ntrainData[numeric_cols] = scaler.fit_transform(trainData[numeric_cols])\n# trainData = scaler.fit_transform(trainData)\n# train = scaler.fit_transform(train)\n# train_labels = scaler.fit_transform(train_labels)\n\ntrainData = trainData.astype('float32')\ntrainData_labels = trainData_labels.astype('float32')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from random import random\n# Train / Validation Split of test data\nfVal_size = 0.20\nseed = int((random()*1000)//1) # 7\nprint(\"random seed is  = \", seed)\nscoring = 'accuracy'\nX = trainData.copy()\nY = trainData_labels.copy()\nX_train,X_val, Y_train, Y_val = model_selection.train_test_split( X, Y,\n                                                                test_size=fVal_size,\n                                                                random_state = seed)\n\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"for nMax in range(150):\n    nECodes = len(Y_train.unique())\n    if nECodes < nEbirdCodes-2: # leave room for nocall not there for now\n        print(\"Running split again to get all birds....Run, ecodes # \", nMax, nECodes)\n        seed = int((random()*1000)//1)\n        X_train,X_val, Y_train, Y_val = model_selection.train_test_split( X, Y, stratify=Y, \n                                                                test_size=fVal_size,\n                                                                random_state = seed)\n        \n#         X_train, X_test, y_train, y_test = train_test_split(X, y, stratify=y, random_state=42)\n    else:\n        print(\"Test split Good, ecodes count # \", nECodes)\n        break\n        \n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# X_train.head(3)\nX_train[:1]\nX_train.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"Y_train.head(1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import xgboost as xgb\n# from xgboost import XGBClassifier\n# from sklearn.metrics import accuracy_score\n\n## Algorithm compare\nmodels = []\n# models.append(('LR', LogisticRegression() ))\n# models.append(('LDA', LinearDiscriminantAnalysis() ))\n# models.append(('KNN', KNeighborsClassifier() ))\nmodels.append(('CART', DecisionTreeClassifier(min_samples_leaf = 10) ))\n# models.append(('NB', GaussianNB() ))\n# models.append(('SVM', SVC(kernel='linear') )) ## Very time intensive\n#  self.model=XGBClassifier(n_estimators=1000,learning_rate=0.02,**params)\n# models.append(('XGB-CL', xgb.XGBClassifier(random_state=1,learning_rate=0.01, max_depth=10, verbosity=100) )) #GradientBoostingClassifier(random_state=0)\n\n# models.append(('GB-CL', GradientBoostingClassifier(random_state=1,learning_rate=0.01, max_depth=10) )) #GradientBoostingClassifier(random_state=0)\n\n# import xgboost as xgb\n# model=xgb.XGBClassifier(random_state=1,learning_rate=0.01)\n# model.fit(x_train, y_train)\n# model.score(x_test,y_test)\n\n# compare models\nresults = []\nnames = []\nbFindBestModel = True\nif bFindBestModel == True:\n    for name, model in models:\n        kfold = model_selection.KFold(n_splits=20, random_state=seed)\n        cv_results = model_selection.cross_val_score(model, X_train.values,\n                                                     Y_train, cv=kfold, scoring=scoring,\n                                                     n_jobs=-1, verbose=20)  # -1 is all processors, v=0 none, 10 is all iterations\n        results.append(cv_results)\n        names.append(name)\n        print(\"\",name,\" :: Mean =\", cv_results.mean(), \", STD =\", cv_results.std() )\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"     CART  :: Mean = 0.08211292076825748 , STD = 0.002850388649308943 - at 70\n\n\n    >  LR  :: Mean = 0.09053031379320882 , STD = 0.004155416885909685 at 80\n    >  LDA  :: Mean = 0.07672898283522037 , STD = 0.003380384083820006\n    >  KNN  :: Mean = 0.07647277369049998 , STD = 0.004118654987786331\n    >  CART  :: Mean = 0.09605785587418034 , STD = 0.004120997742153965\n    >  NB  :: Mean = 0.07365413802698048 , STD = 0.005369852575328259\n    \n      CART  :: Mean = 0.0957646280195982 , STD = 0.0038940212223722973 - at 85\n    \n     CART  :: Mean = 0.09983974688631879 , STD = 0.007691210144059721 - at 87\n     \n     CART  :: Mean = 0.10447520432755628 , STD = 0.010142759083692596 - AT 87.5\n     \n     CART  :: Mean = 0.10235265727277416 , STD = 0.00804816806714718- AT 87.83\n     CART  :: Mean = 0.10802477695100365 , STD = 0.007889066415571452 - RUN 2\n","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"bXGB = False\nif bXGB == True:\n    # importing required libraries\n    from xgboost import XGBClassifier\n#     from sklearn.metrics import accuracy_score\n \n    # seperate the independent and target variable on testing data\n#     test_x = test_data.drop(columns=['Survived'],axis=1)\n#     test_y = test_data['Survived']\n#     X_train,X_val, Y_train, Y_val = model_selection.train_test_split\n    \n    model_xgb = XGBClassifier(random_state=1,learning_rate=0.01, max_depth=10, verbosity=1)\n \n    # fit the model with the training data\n    model_xgb.fit(X_train,Y_train)\n \n \n    # predict the target on the train dataset\n    predict_train = model_xgb.predict(X_train)\n    print('\\nTarget on train data',predict_train) \n \n    # Accuray Score on train dataset\n    accuracy_train = accuracy_score(Y_train,predict_train)\n    print('\\naccuracy_score on train dataset : ', accuracy_train)\n \n    # predict the target on the test dataset\n    predict_test = model_xgb.predict(X_val)\n    print('\\nTarget on test data',predict_test) \n \n    # Accuracy Score on test dataset\n    accuracy_test = accuracy_score(Y_val,predict_test)\n    print('\\naccuracy_score on test dataset : ', accuracy_test)\n    \n# accuracy_score on train dataset :  0.256266850865661\n# accuracy_score on test dataset :  0.08021390374331551","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# linear regression feature importance\nfrom sklearn.datasets import make_regression\nfrom sklearn.linear_model import LinearRegression\nfrom matplotlib import pyplot\n# define dataset\n# X, y = make_regression(n_samples=1280, n_features=27, n_informative=5, random_state=1)\nbBuildModel = False # for submit set to false to save/load\nif bBuildModel == True:\n# define the model svm = svm.SVC(kernel='linear')\n#     model = SVC(kernel='linear', probability=True, verbose=True) # default LR = 0.01\n# model = LinearRegression() #\n    model = DecisionTreeClassifier(min_samples_leaf = 10) #\n    # fit the model\n    model.fit(X_train, Y_train)\n    # save model\n    pickle.dump(model, open(model_filename, 'wb'))\nelse:\n# load the model from disk\n    model = pickle.load(open(model_filename, 'rb'))\n#     result = loaded_model.score(X_test, Y_test)\n\n    \nbBuildModel = True    \nif bBuildModel == True:\n\n#   fit(\n#     x=None, y=None, batch_size=None, epochs=1, verbose=1, callbacks=None,\n#     validation_split=0.0, validation_data=None, shuffle=True, class_weight=None,\n#     sample_weight=None, initial_epoch=0, steps_per_epoch=None,\n#     validation_steps=None, validation_batch_size=None, validation_freq=1,\n#     max_queue_size=10, workers=1, use_multiprocessing=False  )\n    \n# get importance\n# importance = model.coef_ # LinRegr..\n    importance = model.feature_importances_  # CART\n#     importance = abs(model.coef_[0]) # SVM..\n\n\n# summarize feature importance\n    for i,v in enumerate(importance):\n        print('Feature: %0d, Score: %.5f' % (i,v))\n# plot feature importance\n    pyplot.bar([x for x in range(len(importance))], importance)\n    pyplot.show()\n    \n    ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# importance[1]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def perc2Pred(data, fThresh=3.0, nTopBirds=1):\n#     print(data[:5])\n    top_n_pred = np.argsort(data, axis=1)[:,-nTopBirds :]\n    class_labels = model.classes_\n    birdPr = class_labels[top_n_pred][:,0]\n    for index in range(len(birdPr)):\n        fPercentage = data[index,top_n_pred[index]]*100\n#         print(\"Percentage = \", fPercentage)\n        if fPercentage<fThresh: # Theshold\n            birdPr[index] = enumECodes['nocall']\n    return birdPr","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# predData = dbPrep4Predict(bFinderDb)\n# #     SVMpreds = model.predict(predData)\n# SVMpreds = model.predict_proba(predData)\n# SVMpreds2 = perc2Pred(SVMpreds, fThresh=3)#fThresh=5.0, nTopBirds=1\n\n# print(SVMpreds2[:25])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# perc2Pred(SVMpreds,35.0)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"if bBuildModel == True:\n# model = DecisionTreeClassifier() #\n# fit the model\n# model.fit(X_train, Y_train)\n    SVMpreds = model.predict_proba(X_val) # predict with probability\n    SVMperc = perc2Pred(SVMpreds,3.0) # probability cutoff\n# SVMpreds = model.predict(X_val)\n\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"if bBuildModel == True:\n    print(accuracy_score(Y_val,SVMperc ))\n    print(confusion_matrix(Y_val,SVMperc ))\n    print(classification_report(Y_val,SVMperc ))\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# SVMperc = perc2Pred(SVMpreds)\n# SVMperc","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# SVMpreds.shape\n# n=1\n# top_n_pred = np.argsort(SVMpreds, axis=1)[:,-n :]\n# class_labels = model.classes_","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# SVMperc[:20]\n# SVMVals = pd.value_counts(SVMperc, ascending=False) # was SVMpreds\n\n# SVMVals_high = SVMVals[:10]\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# SVMVals","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# print(zip(*SVMVals)[0]) #zip(*tuples)[0]\n\n# summarize predictions\n# for i,v in enumerate(SVMVals):\n#     if v>10:\n#         print('Feature: %0d, Score: %.5f' % (i,v))\n# # plot preds count\n# pyplot.bar([x for x in range(len(SVMVals))], SVMVals)\n\n# pyplot.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Submission\n","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# def fiveSecPredict(data):\n#     topBirds = []\n#     topBirdPreds = []\n    \n#     return topBirds,topBirdPreds","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# def runPredict():\n#     # 3 sites, first 2 by 5 second labels, 3rd by whole file labels\n#     # How about  second intervals, get top 5 predicted birds in each 5 secs\n    \n    \n#     return","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Submission parsing\n\nmaybe use, if sample times given...\n\nSteps:\nGet File, get total seconds\nGet file parsing info\nif not info parse every 5 seconds\n","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"test.head(2)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# def splitLargeFile(large_clip):\n#     clip_arr = []\n#     nPoints = len(large_clip)\n#     fSecFile = nPoints/nSAMPLERATE \n#     secIteration = 5.0\n    \n#     secLastIter = fSecFile % secIteration\n#     nTotIter = math.ceil(fSecFile / secIteration)\n#     if secLastIter < 2.0 and fSecFile > secIteration:\n#             nTotIter = nTotIter-1 # Last iter too short, let's skip it  ### Keep if Submission file??\n#     y_bird_arr = np.empty((int(nTotIter), math.ceil(secIteration*nSAMPLERATE)-1),dtype=float)\n#     for nIter in range(nTotIter):\n#         if fSecFile <= secEnd:\n#             secEnd = fSecFile\n#         audio_bird =  (large_clip[int(secStart*nSAMPLERATE):int(secEnd*nSAMPLERATE)-1]).copy()\n#         clip_arr[nIter]= audio_bird.copy()   ## Building Array of 5 second segments\n        \n#     return clip_arr","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# def make_prediction(sound_clip, birds=dbEbirdCodes):\n#     return np.random.choice(birds)\n\ndef make_prediction(sound_clip, birds=dbEbirdCodes):\n    \n#     addBirdFileToQ(loopSong, idTrain=loopSongID)\n    \n    if np.random.randint(0,9) % 5 == 0:\n        return np.random.choice(birds)\n    return \"nocall\" # near half \"nocall\", so just guessing at ~10%","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"\ndef dbPrep4Predict(data):\n    if len(data) < 1:\n        print(\"ERROR - dbPrep4Predict - empty data array.\")\n        return []\n     # bFinderDb.head(30)\n    numeric_cols = list(set(data.columns) )\n#     data1 = data.loc[:,'primFr':].fillna(0).copy() # 27 columns\n    data1 = data.loc[:,['fFundFreq','dCycle','primSl']].fillna(0).copy() # 3 columns\n\n#     print(data1[:5])\n#     data1[numeric_cols] = scaler.fit_transform(data1[numeric_cols])\n    data1 = scaler.fit_transform(data1)\n#     print(data1)\n    return data1\n\n\n# pred = model.predict(test1)[0]     #make_prediction(sound_clip)\n# print(\"Pred, p = \",[bFinderDb[index]], pred )","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# bFinderDb","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# predData = dbPrep4Predict(bFinderDb)\n# # print(predData[:5])\n# #     SVMpreds = model.predict(predData)\n# SVMpreds = model.predict_proba(predData)\n# # print(SVMpreds[:5])\n# SVMpreds2 = perc2Pred(SVMpreds,30.0)\n# print(SVMpreds2[:5])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# numeric_cols=list(set(bFinderDb.columns) )","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# dbPrep4Predict(bFinderDb)\n                    \n#                     loopSong = train.iloc[int(loopSongID) ]['full_path']\n                   ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"\ndef Run_Predict(test):  ## enter with test = test db to use\n    global dbEbirdCodes\n    global bRUNPARALLEL\n    global nCPUs\n    initRun()\n    global bPREDICTRUN\n    bPREDICTRUN = True\n    global SVMpreds # just for debugging\n    bRUNPARALLEL = False\n    global code\n    global loopSongID\n    loopSongID = 0\n    print(\"Init done, starting filling que....\")\n    bQueing = True\n    startTime = time.time()\n\n    try:  # Comment out top and bottom to allow error propogation\n    # if True:\n        preds = []\n        nCount=0\n        for index, row in test.iterrows():\n            loopSongID = index # same as ID in training\n            nCount +=1\n            bTestPredict = False\n#             if (nCount >= 5) and ( (bSUBMITRUN == False) or (bTestPredict == True) ):\n#                 qToProcess.join()\n#                 print(\"Exit after quick test of 5\")\n#                 continue\n            \n            \n            # Get test row information\n            site = row['site']\n            start_time = row['seconds'] - 5.0\n            row_id = row['row_id']\n            audio_id = row['audio_id']\n            row_path = row['path']\n  \n            print(\"count = #\", index, \", start, end = \", start_time, \", \", row['seconds'], \" ####### \" )\n            qToProcess.join()\n            # Get the test sound clip\n            if site == 'site_1' or site == 'site_2':\n                fDur = 5.0\n                addBirdFileToQ(row_path, idTrain=loopSongID,fStartSec = start_time,fDur = fDur)\n#                 sound_clip = load_test_clip(row_path, start_time)\n                print(\"===== site = \", site, \" - \", (audio_id + '.mp3'), \" - 5 sec\")\n                if nCount < 3:\n                    print(\"+++++ path = \", (TEST_FOLDER + audio_id + '.mp3') )\n            else:\n                addBirdFileToQ(row_path, idTrain=loopSongID,fStartSec = 0.0,fDur = None)\n#                 sound_clip = load_test_clip(TEST_FOLDER + audio_id + '.mp3', 0, duration=None)\n                print(\"site = \", site, \" - \", (audio_id + '.mp3'), \" - No LIMIT\")\n            \n\n\n\n    except Exception as e:\n        preds = pd.read_csv('../input/birdsong-recognition/sample_submission.csv')\n        print(\"EXCEPTION -- count = \", nCount,\" - \", row_id, audio_id)\n        print(\"EXCEPTION  = \",e)\n        predDF = pd.DataFrame(preds, columns=['row_id', 'birds'])\n        bRUNNING = False\n        return predDF\n       \n            \n    # block until all tasks are done\n    qToProcess.join()\n    print('All work completed')\n    bRUNNING = False\n    \n    if bBuildModel == True:\n        predData = dbPrep4Predict(bFinderDb)\n#     SVMpreds = model.predict(predData)\n        SVMpreds = model.predict_proba(predData)\n        SVMpreds = perc2Pred(SVMpreds, fThresh=5.0)#fThresh=5.0, nTopBirds=1\n        print(SVMpreds[:15])\n    \n        predBirds=[]\n        nPreds = len(SVMpreds)\n        for index in range(nPreds):\n            predBirds = np.append(predBirds,enumECodesRev[int(SVMpreds[index])])\n\n        print(\"Shape compare test vs predBirds = \", test.shape[0],\" vs \", predBirds.shape[0])\n    else:\n        predBirds=[]\n        nPreds = len(bFinderDb)\n        print(\"No BUILD output....\")\n        strPreds=\"\"\n        for index,row in bFinderDb.iterrows():\n            if row['ecode'] == enumECodes['nocall']:\n                predBirds = np.append(predBirds,'nocall')\n            else:  \n#                 predBirds = np.append(predBirds,'amecro')\n                if int(row['nSegm']) < 1: # for now ONLY catch first bird segm. in area 3\n                    predBirds = np.append(predBirds,enumECodesRev[ row['ecode'] ]) # Reverse lookup string\n        \n#         for index in range(nPreds):\n#             predBirds = np.append(predBirds,enumECodesRev[int(SVMpreds[index])])\n            \n\n#         print(\"Shape compare test vs predBirds = \", test.shape[0],\" vs \", predBirds.shape[0])\n        \n    preds = pd.DataFrame({'row_id':test['row_id'][:nPreds], 'birds': predBirds[:nPreds] } )\n    \n    predDF = pd.DataFrame(preds, columns=['row_id', 'birds'])\n\n    \n    endTime = time.time()\n    print(\"Total time = {:.4f}\".format(endTime-startTime))\n    print(\"Total time = \", (endTime-startTime))\n    return predDF","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"bTestPredict = False\n# bPREDICTRUN = True\nif bPREDICTRUN == True or bTestPredict == True:\n    preds = Run_Predict(test)\n    saveData(bFinderDb, strFileName = \"Bproj_data2\")\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"bFinderDb.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"if bPREDICTRUN == True or bTestPredict == True:\n    preds.head(-30)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# preds\n# bFinderDb[:25]\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"bFinderDb[-30:]\n\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"\n# bFinderDb[-50:]\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"\n# predData = dbPrep4Predict(bFinderDb)\n# #     SVMpreds = model.predict(predData)\n# SVMpreds = model.predict_proba(predData)\n# SVMpreds = perc2Pred(SVMpreds, fThresh=1.99)#fThresh=5.0, nTopBirds=1\n\n# print(SVMpreds[:5])\n    \n# predBirds=[]\n# for index in range(len(test)):\n#     predBirds = np.append(predBirds,enumECodesRev[int(SVMpreds[index])])\n\n# preds = pd.DataFrame({'row_id':test['row_id'], 'birds': predBirds } )\n    \n# predDF = pd.DataFrame(preds, columns=['row_id', 'birds'])\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"\n# frame = { 'Author': auth_series, 'Article': article_series }  \n# result = pd.DataFrame(frame) \n# predBirds=[]\n# for index in range(len(test)):\n#     predBirds = np.append(predBirds,enumECodesRev[int(SVMpreds[index])])\n\n# preds = pd.DataFrame({'row_id':test['row_id'], 'birds': predBirds } )\n# preds.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# key = next(key for key, value in dd.items() if value == 'value')\n\n# predBirds=[]\n# for index in range(len(SVMpreds)):\n#     predBirds = np.append(predBirds,enumECodesRev[int(SVMpreds[index])])\n\n\n# # SVMpreds = int(value)   for value in SVMpreds[:10]\n# predBirds","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"\n# def getBFile():\n#     return\n\n# def getBFileSplits():\n#     return\n\n# def buildSplitLabels():\n#     return\n\n# if sample.site in ['site_1', 'site_2']:\n#     start, end = sr * (sample.seconds - 5), sr * (sample.seconds)\n#     y = y[int(start):int(end)]\n# else:\n#     start, end = sr * (0), sr * (5)\n#     y = y[int(start):int(end)]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# #  train , test is our sets here\n\n# train = train_all.loc[:,'primFr':].fillna(0)\n# # train = train_full1.loc[:,'primFr':'dCycle'] \n# train_labels = train_all['ecode'].fillna(0)\n\n\n# numeric_cols = list(set(train.columns) )\n# # train[numeric_cols] = train[numeric_cols].fillna(0)\n# print('scaling numeric columns')\n\n# scaler = StandardScaler()\n# train[numeric_cols] = scaler.fit_transform(train[numeric_cols])\n# # train = scaler.fit_transform(train)\n# # train_labels = scaler.fit_transform(train_labels)\n\n# train = train.astype('float32')\n# train_labels = train_labels.astype('float32')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"if bPREDICTRUN == True:\n    saveSubmission(preds)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"if bPREDICTRUN == True:\n    preds.head(20)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_all_filtered2.sort_values(['ecode'], ascending=True, inplace=True)\ntrain_all_filtered2.head(25)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"groupSummary = train_all_filtered2.groupby(by=[\"ecode\"])\ngroupSummary.head(3)\n\nnCnt = 0\nfor name, group in groupSummary:\n    nCnt +=1\n    if nCnt % 100 == 0:\n        print(\"name, group in groupSummary = \", name,\" - \", enumECodesRev[name])  #, group)\n        bPrint = True\n  \n        for row_index, row in group.iterrows():\n            if bPrint:\n                print(row[3])\n                print(row_index)\n                bPrint = False\n\n# df = train_all_filtered2.groupby('ecode').agg(lambda x: ','.join(x))\n# for name in df.index:\n#     print(name)\n#     print(df.loc[name])\n    \n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#PEP-8 naming conventions\n# variable_one_name = 1  # same for functions\n# class CamelCaseStyle(object):  # same for TypeVariables\n# This is PEP 8 conforming comment\n\n\"\"\"Multiline Function descriptions.  First line is summary. \n\nThis function takes a pandas dataframe with time series as the index, \nand calculates the total sum, aggregated by month. \n\"\"\"\n\n# CONSTANT_ONE = 99\n# def f(x): return 2*x   insead of ...   f = lambda x: 2*x   # single line functions","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def app_imports(): # call in public space, regardless will import to global space\n    import numpy as np \n    import pandas as pd \n\n    import os\n\n    import IPython.display as ipd\n    pd.set_option('max_columns', 35)\n    import matplotlib.pyplot as plt\n    %matplotlib inline\n    import seaborn as sns\n    import pickle # serializing models  \n    # save model\n    # pickle.dump(model, open(filename, 'wb'))\n    # load the model from disk\n    # loaded_model = pickle.load(open(filename, 'rb'))\n    # result = loaded_model.score(X_test, Y_test)\n\n    import shutil\n    # shutil.copy2('/src/dir/file.ext', '/dst/dir/newname.ext') # complete target filename given\n    # shutil.copy2('/src/file.ext', '/dst/dir') # target filename is /dst/dir/file.ext\n    # Imports\n\n\n    import matplotlib.image as mpimg\n    from matplotlib.offsetbox import AnnotationBbox, OffsetImage\n\n    # Map 1 library\n    import plotly.express as px\n\n    # Map 2 libraries\n    import descartes\n    import geopandas as gpd\n    from shapely.geometry import Point, Polygon\n\n    #install numba==0.48\n    # above is hack to not hit 0.5 removal of decorators\n\n    import numba\n    #==0.48\n    import numba.decorators\n\n    import sklearn\n\n    import warnings\n    warnings.filterwarnings('ignore')\n    \n    # Sound Imports\n    import librosa\n    import librosa.display\n    from joblib import Parallel, delayed, cpu_count\n    import time, math # for display type use ME\n\n\n    import pylab\n\n    import torchaudio\n    from scipy import signal\n\n    import gc\n    from path import Path\n    \n    # Threading\n    import threading, queue\n    \n    return None\n    ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def onlyNAmerica(data, data_info=[]): # top 3 are N America, will filter out of data, based on data_info (in case different)\n    if len(data_info)<1:\n        data_info = data\n    top_3 = list(data_info['country'].value_counts().head(3).reset_index()['index'])\n    dInfo_top_3 = data_info[data_info['country'].isin(top_3)]\n    data_top_3 = data[data['filename'].isin(dInfo_top_3['filename']) ]  #logical_not = data[~data\n    return data_top_3","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def add_constants(class_instance):\n    class_instance.nSAMPLERATE = 22050\n    class_instance.nMAXSONGITER = 5\n    return None\n    ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def add_variables(class_instance):\n    class_instance.bRUNNING = True # was bRUNNING\n    class_instance.bSUBMITRUN = False\n    class_instance.bPREDICTRUN = False ## Set to False for Data, True for Predict runs (automatically)\n    class_instance.bFASTRUN = True\n    class_instance.bRUNPARALLEL = True\n\n    class_instance.bFILTER = True # butter filter\n    \n    return None","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def add_files(class_instance):\n    class_instance.bRUNNING = True # was bRUNNING\n    class_instance.bSUBMITRUN = False\n    class_instance.bPREDICTRUN = False ## Set to False for Data, True for Predict runs (automatically)\n    class_instance.bFASTRUN = True\n    class_instance.bRUNPARALLEL = True\n\n    class_instance.bFILTER = True # butter filter\n    \n    \n    class_instance.model_filename = 'BFinder_model.sav'  # if first run, copy file to output folder (for path consistency..)\n    if os.path.isfile(class_instance.model_filename):\n        print(\"Model File already exists in output.\")\n    else:\n        shutil.copy2('/kaggle/input/zobirdsongdata3/BFinder_model.sav', 'BFinder_model.sav') # target filename is /dst/dir/file.ext\n        print(\"Copying from saved model to output folder\")\n    # Step on file in output if not saved\n\n    class_instance.TEST = Path(\"../input/birdsong-recognition/test_audio\").exists()\n    class_instance.bSUBMITRUN = class_instance.TEST\n    if class_instance.TEST:\n        class_instance.DATA_DIR = str(Path(\"../input/birdsong-recognition\"))\n        print(\"SUBMIT PATH EXISTS\")\n    else:\n        # dataset created by @shonenkov, thanks!\n        class_instance.DATA_DIR = str(Path(\"../input/birdcall-check\"))\n        print(\"NON-SUBMIT - Using Alternative PATH\")\n\n    print(\"DATA_DIR (Dynamic) = \",class_instance.DATA_DIR)\n\n\n    class_instance.TEST_FOLDER = class_instance.DATA_DIR + '/test_audio/'\n    class_instance.test = pd.read_csv(class_instance.DATA_DIR + \"/test.csv\")\n\n    class_instance.test.head()\n    \n    \n    \n    \n    \n    return None","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"class BirdFinder(object):\n    # Class Variables (public)\n    source_data = []\n    \n    # Class Constants.  - outside class prefix like: instance.variable\n#     bRUNNING = True\n    \n    def __init__(self):\n        self.source_data = []\n        self.private_data = [] # class only access\n        add_constants(self)\n        add_variables(self)\n        add_files(self)\n\n    def add(self, x):\n        self.source_data.append(x)\n\n    def addtwice(self, x):\n        self.add(x)\n        self.add(x)\n        \n    ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"app_imports()\n\n\n# Creates new instance.\nmy_finder = BirdFinder()\n# Calls its attribute.\nmy_finder.addtwice([5])\nprint(my_finder.source_data)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"my_finder.nMAXSONGITER","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}