{"cells":[{"metadata":{"_uuid":"efb8c0912d4250e56b466bbace3f6a14ecaa42d6"},"cell_type":"markdown","source":"## After spending some time looking at other people's work, I note\n- The initial label for this approach, 'Fast Feature Extractor,' was a misnomer\n- Other people's approaches work better (inital score training GBM with these features was 1.958)\n\n### HOWEVER\n- That doesn't mean there's nothing of value here\n- Most of the public kernels take very similar approaches to one another\n- Reading an outsider's approach may trigger some new ideas for subject matter experts"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"#The standard imports Kaggle give you when you start a kernel\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list the files in the input directory\n\nimport os\nprint(os.listdir(\"../input\"))\n\n# Any results you write to the current directory are saved as output.","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f314b3fe3fb6f3f6679f407189e10a4ed7c07af6"},"cell_type":"markdown","source":"**I need a few more libraries**"},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"import warnings\nfrom matplotlib import pyplot as plt\nimport copy\nimport scipy.stats as ss\n#from astropy.time import Time\n#import fbprophet as fbp\n#import FATS\n#import cesium\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e3948913e9cb79934ba0bc460cfd5c9522f22a41"},"cell_type":"markdown","source":"**Let's grab the training data**\n- If we can't get a feature table for the training data in ~15 minutes we'll be in trouble with the test data\n- There are roughly 500 times more objects in the test set"},{"metadata":{"trusted":true,"_uuid":"1b9f42ab2195d51e7b1622c30fdaecc3b424b77b"},"cell_type":"code","source":"print('Getting Base (meta) dataFrame')\n#bdf=pd.read_csv('../input/test_set_metadata.csv')\nbdf=pd.read_csv('../input/training_set_metadata.csv')\n\nprint(bdf.shape)\n#print('Postponing getting raw (lightcurve) dataFrame')\nrdf=pd.read_csv('../input/training_set.csv')\n#print(rdf.shape)\nprint(rdf.shape)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"cde22efb53961fd55f0de3ed90f846ba2d6e763d"},"cell_type":"markdown","source":"## This won't be used for the test data\n- but the principle will\n- 2 and 5 were chosen because they are far apart on the spectrum and they have a lot of data\n- 0 and 1 have significantly less data in the Test data set"},{"metadata":{"trusted":true,"_uuid":"b45518da618a85ce061cc859bc2f42777f484a17"},"cell_type":"code","source":"lep=2\nhep=5\n# originally planned on 2, 4 based on DDF data\n# based on all data going with 2, 5\n\ndef filterRawDf(ordf, lowEngPb=lep, highEngPb=hep):\n #frdf=rdf[rdf['passband'] in [2,5]]\n rdf=copy.deepcopy(ordf)\n filterLow = rdf.loc[:,'passband']==lowEngPb\n filterHigh = rdf.loc[:,'passband']==highEngPb\n filterPb = filterLow | filterHigh\n frdf=rdf.loc[filterPb,:]\n return frdf\n\nfrdf = filterRawDf(rdf)\nfrdf.shape\n\n#t=Time(frdf.loc[:,'mjd'])\n#print(t.isot)\n#frdf=frdf.rename(columns={'mjd':'ds', 'flux':'y'})","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"17c535d8d645166b5e2b16cdc20853bba2427983"},"cell_type":"markdown","source":"## This method won't be used with the test data, but it's conventient here\n- the test data will use the 'fast_test_set_reading' method"},{"metadata":{"trusted":true,"_uuid":"2c4a66481ce26470b800e1d509bf117276c8d4db"},"cell_type":"code","source":"#get light curve data for one object_id from the raw data\n#this can be used with either rdf or filtered raw data (frdf)\ndef getLCDF(ordf, objid, show=False):\n rdf=copy.deepcopy(ordf)\n lcdf=rdf[rdf['object_id']==objid]\n if show:\n plt.plot(lcdf.loc[:,'mjd'],lcdf.loc[:,'flux'])\n plt.show()\n\n \n return lcdf\n \nelcdf=getLCDF(frdf,615, show=True)\nprint(elcdf.shape)\n#print(frdf.shape)\nelcdf=getLCDF(frdf, 1019335, show=True)\nprint(elcdf.shape)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"4bb87049e0d7411efecabc3da1a1c7a15aa0ea8b"},"cell_type":"markdown","source":"### We need to divide the light curve\n- we want to see if beginning, middle, end are all the same (permanent objects)\n- or different (transitory events)\n- we note the ddf=1 and ddf=0 curves have different sampling methodologies\n"},{"metadata":{"trusted":true,"_uuid":"7d07c90b1e93310df69bbacb43cc2a9e9a6a9327"},"cell_type":"code","source":"def divideLcdf(elcdf, ddf, lep=2, hep=5):\n lcdf=copy.deepcopy(elcdf)\n #this simple date cutting works on the ddf objects\n if ddf:\n minDate=np.min(elcdf.loc[:,'mjd'])\n maxDate=np.max(elcdf.loc[:,'mjd'])\n\n halfPoint=np.average([minDate, maxDate])\n firstCut=np.average([minDate, halfPoint])\n secondCut=np.average([halfPoint, maxDate])\n minDate=np.min(elcdf.loc[:,'mjd'])\n maxDate=np.max(elcdf.loc[:,'mjd'])\n\n halfPoint=np.average([minDate, maxDate])\n firstCut=np.average([minDate, halfPoint])\n secondCut=np.average([halfPoint, maxDate])\n\n #early\n efilter=elcdf.loc[:,'mjd']<=firstCut\n #late\n lfilter=elcdf.loc[:,'mjd']>=secondCut\n #mid\n mfilter=(efilter | lfilter)==False\n \n edf=elcdf.loc[efilter]\n mdf=elcdf.loc[mfilter]\n ldf=elcdf.loc[lfilter]\n \n ledf = edf[edf['passband']==lep]\n hedf = edf[edf['passband']==hep]\n lmdf = mdf[mdf['passband']==lep]\n hmdf = mdf[mdf['passband']==hep]\n lldf = ldf[ldf['passband']==lep]\n hldf = ldf[ldf['passband']==hep]\n \n #using the datecutting method often leads to zero population sizes with non-ddf objects\n else:\n \n lowdf=elcdf[elcdf['passband']==lep]\n highdf=elcdf[elcdf['passband']==hep]\n lenLow=lowdf.shape[0]\n lenHigh=highdf.shape[0]\n \n minSizeLow = int(lenLow / 3)\n minSizeHigh = int(lenHigh / 3)\n \n lldf=lowdf.nlargest(minSizeLow, 'mjd')\n hldf=highdf.nlargest(minSizeHigh, 'mjd')\n ledf=lowdf.nsmallest(minSizeLow, 'mjd')\n hedf=highdf.nsmallest(minSizeHigh, 'mjd')\n lmdf=lowdf.nlargest(lenLow-minSizeLow, 'mjd').nsmallest(lenLow-2*minSizeLow, 'mjd')\n hmdf=highdf.nlargest(lenHigh-minSizeHigh, 'mjd').nsmallest(lenHigh-2*minSizeHigh, 'mjd')\n \n return ledf, hedf, lmdf, hmdf, lldf, hldf\n\nledf, hedf, lmdf, hmdf, lldf, hldf=divideLcdf(elcdf, 0) \nprint(ledf.shape)\nprint(lmdf.shape)\nprint(lldf.shape)\nprint(hedf.shape)\nprint(hmdf.shape)\nprint(hldf.shape)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c88fe18371338c396de2f614186d5836d49a56f6"},"cell_type":"markdown","source":"### This method will get features on subpopulations\n- When the subpopulations for both low energy and high energy bands overlap each other, we have a steady state object, more on this later\n- This has evolved from the first draft (eliminating percent within different sigma bands)"},{"metadata":{"trusted":true,"_uuid":"2dc31dcb31169f93c627365cce67cd666437a93c"},"cell_type":"code","source":"def getSubPopFeats(pbdf, outSig=3.0):\n \n average=np.average(pbdf.loc[:,'flux'])\n median=np.median(pbdf.loc[:,'flux'])\n stdev=np.std(pbdf.loc[:,'flux'])\n maxflux=np.max(pbdf.loc[:,'flux'])\n minflux=np.min(pbdf.loc[:,'flux'])\n stdflerr=np.std(pbdf.loc[:,'flux_err'])\n medflerr=np.median(pbdf.loc[:,'flux_err'])\n \n #We want a means to extract the rate of decay or rise from minima or maxima\n #This is grabbing within the population\n #We also will look between populations\n maxmjd=np.max(pbdf[pbdf['flux']==maxflux].loc[:,'mjd'])\n minmjd=np.max(pbdf[pbdf['flux']==minflux].loc[:,'mjd'])\n \n #at what date does the max occur?\n aftmaxdf=pbdf[pbdf['mjd']>maxmjd]\n \n #if there are data points after the max, what is the value and date of the lowest?\n if aftmaxdf.shape[0]>0:\n minaft=np.min(aftmaxdf.loc[:,'flux'])\n aftminmjd=np.min(aftmaxdf[aftmaxdf['flux']==minaft].loc[:,'mjd'])\n #(val at t0 - val at t1) / (t0 - t1) sb neg\n decaySlope=(maxflux-minaft)/(maxmjd-aftminmjd)\n \n else:\n decaySlope=0\n \n aftmindf=pbdf[pbdf['mjd']0:\n maxaft=np.max(aftmindf.loc[:,'flux'])\n aftmaxmjd=np.max(aftmindf[aftmindf['flux']==maxaft].loc[:,'mjd'])\n #(val at t0 - val at t1) / (t0 - t1) sb pos\n riseSlope=(minflux - maxaft)/(aftmaxmjd-minmjd)\n \n else:\n riseSlope=0\n \n return average, stdev, median, medflerr, stdflerr, maxflux, \\\n maxmjd, decaySlope, minflux, minmjd, riseSlope\n\na,b,c,d,e,f,g, h,i,j,k=getSubPopFeats(hmdf)\nprint(a)\nprint(b)\nprint(c)\nprint(d)\nprint(e)\nprint(f)\nprint(g)\nprint(h)\nprint(i)\nprint(j)\nprint(k)\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"82761084e63308c4429bf8d15b4672cb449937fd"},"cell_type":"markdown","source":"### The processing isn't done, but we will do as much as possible array style later\n- TBD how much slowdown this change causes. I am now returning 12 instead of 7\n- Want to get decay rates within subsection in case peak at last subsection"},{"metadata":{"trusted":true,"_uuid":"9521cc5bc99a95e550221b31a45cc967b99f4afc"},"cell_type":"code","source":"def processLc(objid, elcdf, ddf, lep=2, hep=5):\n \n lcdf=copy.deepcopy(elcdf)\n \n #feature borrowed from Grzegorz Sionkowski (../sionek)\n #dt[detected==1, mjd_diff:=max(mjd)-min(mjd), by=object_id]\n #detectMjds=elcdf[elcdf['detected']==1].loc[:,'mjd']\n #deltaDetect=np.max(detectMjds) - np.min(detectMjds)\n \n #divide the incoming light curve to 6 subpopulations\n ledf, hedf, lmdf, hmdf, lldf, hldf=divideLcdf(lcdf, ddf,lep=lep, hep=hep)\n #return average, stdev, median, medflerr, stdflerr, maxflux, \\\n # maxmjd, decayslope, minflux, minmjd, riseSlope\n \n leavg, lestd, lemed, lemfl, lesfl, lemax, lemxd, ledsl, lemin, lemnd, lersl=getSubPopFeats(ledf)\n heavg, hestd, hemed, hemfl, hesfl, hemax, hemxd, hedsl, hemin, hemnd, hersl=getSubPopFeats(hedf)\n lmavg, lmstd, lmmed, lmmfl, lmsfl, lmmax, lmmxd, lmdsl, lmmin, lmmnd, lmrsl=getSubPopFeats(lmdf)\n hmavg, hmstd, hmmed, hmmfl, hmsfl, hmmax, hmmxd, hmdsl, hmmin, hmmnd, hmrsl=getSubPopFeats(hmdf)\n llavg, llstd, llmed, llmfl, llsfl, llmax, llmxd, lldsl, llmin, llmnd, llrsl=getSubPopFeats(lldf)\n hlavg, hlstd, hlmed, hlmfl, hlsfl, hlmax, hlmxd, hldsl, hlmin, hlmnd, hlrsl=getSubPopFeats(hldf)\n \n \n feats= [objid, leavg, lestd, lemed, lemfl, lesfl, lemax, \n lemxd, ledsl, lemin, lemnd, lersl,\n heavg, hestd, hemed, hemfl, hesfl, hemax, hemxd,\n hedsl, hemin, hemnd, hersl,\n lmavg, lmstd, lmmed, lmmfl, lmsfl, lmmax, lmmxd,\n lmdsl, lmmin, lmmnd, lmrsl,\n hmavg, hmstd, hmmed, hmmfl, hmsfl, hmmax, hmmxd, \n hmdsl, hmmin, hmmnd, hmrsl,\n llavg, llstd, llmed, llmfl, llsfl, llmax, llmxd, \n lldsl, llmin, llmnd, llrsl,\n hlavg, hlstd, hlmed, hlmfl, hlsfl, hlmax, hlmxd, \n hldsl, hlmin, hlmnd, hlrsl]\n \n return feats\n\nfeats=processLc(1019335, elcdf, 0)\n\nprint(feats)\n ","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"2347ad33d1eea2d53ee2e956bd7a506d056dc29d"},"cell_type":"markdown","source":"### There is more processing to do, but it should be done array-style\n- should be able to induce amplitude and frequency from perHalfSigma, percOneSigma, percTwoSigma\n- should know if outliers are high or low based on median average comparison\n- etc"},{"metadata":{"trusted":true,"_uuid":"40bc746233243eca13e48d2f70df82f0b419aa55"},"cell_type":"markdown","source":"\n# NOTE - [THIS METHOD](https://www.kaggle.com/jimpsull/fast-test-set-reading-merged-with-fast-extractor) is MUCH faster\n- but here I'm just going to use the loaded training set\n- will need to merge fast test set reading with this extractor"},{"metadata":{"trusted":true,"_uuid":"2bc0413c1ec907050334690f3e477d8ea333e41f"},"cell_type":"code","source":"from io import StringIO\nfrom csv import writer \nimport time\n\ndef writeAChunk(firstRecord, lastRecord, bdf, frdf, statusFreq=500):\n output = StringIO()\n csv_writer = writer(output)\n\n fdf=pd.DataFrame(columns=['objid', 'leavg', 'lestd', 'lemed', 'lemfl', 'lesfl', 'lemax', \n 'lemxd', 'ledsl', 'lemin', 'lemnd', 'lersl',\n 'heavg', 'hestd', 'hemed', 'hemfl', 'hesfl', 'hemax', 'hemxd',\n 'hedsl', 'hemin', 'hemnd', 'hersl',\n 'lmavg', 'lmstd', 'lmmed', 'lmmfl', 'lmsfl', 'lmmax', 'lmmxd',\n 'lmdsl', 'lmmin', 'lmmnd', 'lmrsl',\n 'hmavg', 'hmstd', 'hmmed', 'hmmfl', 'hmsfl', 'hmmax', 'hmmxd', \n 'hmdsl', 'hmmin', 'hmmnd', 'hmrsl',\n 'llavg', 'llstd', 'llmed', 'llmfl', 'llsfl', 'llmax', 'llmxd', \n 'lldsl', 'llmin', 'llmnd', 'llrsl',\n 'hlavg', 'hlstd', 'hlmed', 'hlmfl', 'hlsfl', 'hlmax', 'hlmxd', \n 'hldsl', 'hlmin', 'hlmnd', 'hlrsl'])\n\n theColumns=fdf.columns\n \n csv_writer.writerow(theColumns)\n started=time.time()\n for rindex in range(firstRecord, lastRecord):\n #if you want to monitor progress\n #ddf 18 sec per 100 on my macAir\n #non ddf 25 sec per 100 on my macAir\n if rindex%statusFreq==(statusFreq-1):\n print(rindex)\n print(\"Processing took {:6.4f} secs, records = {}\".format((time.time() - started), statusFreq))\n started=time.time()\n #fdf=pd.merge(fdf, tdf, on='key')\n objid = bdf.loc[rindex,'object_id']\n ddf=bdf.loc[rindex,'ddf']==1\n #ig=bdf.loc[rindex,'hostgal_specz']==0\n lcdf=getLCDF(frdf, objid)\n feats=processLc(objid, lcdf, ddf)\n #fdf.loc[rindex,:]=feats\n csv_writer.writerow(feats)\n\n output.seek(0) # we need to get back to the start of the BytesIO\n chdf = pd.read_csv(output)\n chdf.columns=theColumns\n \n return chdf\n\ntheColumns=['objid', 'leavg', 'lestd', 'lemed', 'lemfl', 'lesfl', 'lemax', \n 'lemxd', 'ledsl', 'lemin', 'lemnd', 'lersl',\n 'heavg', 'hestd', 'hemed', 'hemfl', 'hesfl', 'hemax', 'hemxd',\n 'hedsl', 'hemin', 'hemnd', 'hersl',\n 'lmavg', 'lmstd', 'lmmed', 'lmmfl', 'lmsfl', 'lmmax', 'lmmxd',\n 'lmdsl', 'lmmin', 'lmmnd', 'lmrsl',\n 'hmavg', 'hmstd', 'hmmed', 'hmmfl', 'hmsfl', 'hmmax', 'hmmxd', \n 'hmdsl', 'hmmin', 'hmmnd', 'hmrsl',\n 'llavg', 'llstd', 'llmed', 'llmfl', 'llsfl', 'llmax', 'llmxd', \n 'lldsl', 'llmin', 'llmnd', 'llrsl',\n 'hlavg', 'hlstd', 'hlmed', 'hlmfl', 'hlsfl', 'hlmax', 'hlmxd', \n 'hldsl', 'hlmin', 'hlmnd', 'hlrsl']\n\nfdf=pd.DataFrame(columns=theColumns)\nchunksize=2616\nfirstLoop=0\nlastLoop=3\nloops=lastLoop-firstLoop\nveryFirstRow=firstLoop*chunksize\nveryLastRow=lastLoop*chunksize-1\nfor i in range(firstLoop, lastLoop):\n startRow=i*chunksize\n stopRow=(i+1)*chunksize\n chdf=writeAChunk(startRow, stopRow, bdf, frdf, statusFreq=int(chunksize/2))\n fdf= pd.concat([fdf, chdf])\n print(fdf.shape)\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"684babdfa5ac7c9afdb06bcd692253503fffb2d1"},"cell_type":"markdown","source":"### Append the features to the base dataFrame\n- I know there's a more pythonic way to do it, but this was quick and I couldn't get it to work"},{"metadata":{"trusted":true,"_uuid":"ef2665e792f7bcb15191af5ea7c908fc1a2e44ef"},"cell_type":"code","source":"\nfdf=fdf.rename({'objid':'object_id'},axis=1)\nbdf.loc[:,'object_id']=bdf.loc[:,'object_id'].astype(str)\nfdf.loc[:,'object_id']=fdf.loc[:,'object_id'].astype(str)\n#DataFrame.join(other, on=None, how='left', lsuffix='', rsuffix='', sort=False)[source]¶\nmdf=bdf.merge(fdf, sort=False)\nprint(mdf.shape)\nmdf.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"273a7bb7800d24181aa9580a297fcb67adf90f23"},"cell_type":"markdown","source":"### Test for transitory objects\n- what I'm doing here is a psuedo-box-plot without needing the entire distributions\n- I'm looking for populations that don't overlap one another\n- transitory events could be a downslope, and upslope, or and updown (and all be the same type of event)\n- I'll add this processing at a later revision"},{"metadata":{"_uuid":"14cbedc0b3071e9fb1e6840552022c2bf60c1d2f"},"cell_type":"markdown","source":"## Pseudo box-plot\n- I'm trying to see if the populations overlap\n- I'm saying if each population has a one sigma cushion around its average and it doesn't overlap either of the other - population cushions then its an outlier"},{"metadata":{"trusted":true,"_uuid":"5f9d5e96dc997aee7f8afeca0ba5bb3802c6d2bc"},"cell_type":"code","source":"def testForOutlier(bdf, energy='high', sigmas=1.0):\n \n if energy=='high':\n valCols=['heavg', 'hmavg', 'hlavg']\n sigCols=['hestd', 'hmstd', 'hlstd']\n else:\n valCols=['leavg', 'lmavg', 'llavg']\n sigCols=['lestd', 'lmstd', 'llstd']\n \n fdf=copy.deepcopy(bdf)\n \n\n \n fdf.loc[:,energy + 'Energy_transitory_' + str(round(sigmas,1)) + '_TF']=False\n for i in range(len(valCols)):\n fdf.loc[:,'min' + str(valCols[i])] = fdf.loc[:,valCols[i]] - sigmas*fdf.loc[:,sigCols[i]]\n \n fdf.loc[:,'max' + str(valCols[i])] = fdf.loc[:,valCols[i]] + sigmas*fdf.loc[:,sigCols[i]]\n \n for i in range(len(valCols)):\n #fdf.loc[:,'earlySet']=range(fdf.loc[:,'minX100' + str(valCols[0])],fdf.loc[:, 'maxX100' + str(valCols[0])])\n #earlyMaxLessThanMedMin\n for j in range(len(valCols)):\n if j!=i:\n \n maxFailsOverlap=fdf.loc[:,'max' + str(valCols[i])]fdf.loc[:,'max' + str(valCols[j])]\n theValue= (fdf.loc[:,energy + 'Energy_transitory_' + str(round(sigmas,1)) + '_TF'] | minFailsOverlap | maxFailsOverlap)\n #theValue=theValue.astype(str)\n fdf.loc[:,energy + 'Energy_transitory_' + str(round(sigmas,1)) + '_TF']=theValue\n #fdf.loc[:,energy + '_' + str(valCols[i]) + '_' + str(valCols[j])] = str(theValue) + \\\n #+ '_' + str(maxFailsOverlap) + '_'+ str(minFailsOverlap)\n for i in range(len(valCols)):\n fdf=fdf.drop('min' + str(valCols[i]), axis=1)\n fdf=fdf.drop('max' + str(valCols[i]), axis=1)\n \n return fdf\n\nenergy='high'\nsigmas=1.0\nfdf=testForOutlier(mdf)\nfdf.shape\nprint(fdf.loc[:,energy + 'Energy_transitory_' + str(round(sigmas,1)) + '_TF'].sum())\n\nsigmas=1.5\nfdf=testForOutlier(fdf, energy=energy, sigmas=sigmas)\nprint(fdf.loc[:,energy + 'Energy_transitory_' + str(round(sigmas,1)) + '_TF'].sum())\n\nenergy='low'\nsigmas=1.0\nfdf=testForOutlier(fdf, energy=energy, sigmas=sigmas)\nprint(fdf.loc[:,energy + 'Energy_transitory_' + str(round(sigmas,1)) + '_TF'].sum())\n\nsigmas=1.5\nfdf=testForOutlier(fdf, energy=energy, sigmas=sigmas)\nprint(fdf.loc[:,energy + 'Energy_transitory_' + str(round(sigmas,1)) + '_TF'].sum())\n\nprint(fdf.shape)\nfdf.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c588dc309eb41bc252127c7619bae9a822265b66"},"cell_type":"markdown","source":"## Consolidate four booleans into one score - arbitrary scoring choice¶\n- The scoring dictionary is a potential tuning opportunity if this feature shows some power\n- 6,7,8 are all situations where both passbands show some outlier tendencies\n- 3 is where one passband shows outlier tendencies. Reasoning explained more in comments below"},{"metadata":{"trusted":true,"_uuid":"e4e58b782fe7e59ea31c62a48fcc2dd6ddfd5632"},"cell_type":"code","source":"fdf.loc[:,'outlierString']=fdf.loc[:,'highEnergy_transitory_1.5_TF'].astype(str) + \\\n fdf.loc[:,'highEnergy_transitory_1.0_TF'].astype(str) + \\\n fdf.loc[:,'lowEnergy_transitory_1.5_TF'].astype(str) + \\\n fdf.loc[:,'lowEnergy_transitory_1.0_TF'].astype(str)\n\n\ndef getOutlierScore(row):\n tdict={'TrueTrueTrueTrue':8, 'FalseTrueTrueTrue':7, 'TrueTrueFalseTrue':7,\n 'FalseTrueFalseTrue':6, 'FalseFalseTrueTrue':3, 'TrueTrueFalseFalse':3,\n 'FalseTrueFalseFalse':3, 'FalseFalseFalseTrue':3, 'FalseFalseFalseFalse':0}\n return tdict[row['outlierString']]\n\nfdf['outlierScore']=fdf.apply(getOutlierScore, axis=1)\n \nfdf=fdf.drop('outlierString', axis=1)\n\n#fdf.to_csv('fastestFeatureTableWithTransitoryFlags.csv')\nprint(fdf.shape)\nprint(fdf.columns)\nprint(np.average(fdf.loc[:,'outlierScore']))\nprint(np.min(fdf.loc[:,'outlierScore']))\nprint(np.max(fdf.loc[:,'outlierScore']))\nprint(np.median(fdf.loc[:,'outlierScore']))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d5b25b2600672f0e310af89bc45d2821f9ed0b56"},"cell_type":"markdown","source":"## Rate of decay\n- If outlierScore is zero we only care about the intraPop slopes and its amplitude related\n- If it is non-zero we want to see "},{"metadata":{"_uuid":"99c2445bfc164343f3bb743c92935fcc5b4b1a15","trusted":true},"cell_type":"code","source":"fdf['hipd']=0\nfdf['hipr']=0\nfdf['htpd']=0\nfdf['htpr']=0\n\nfdf['lipd']=0\nfdf['lipr']=0\nfdf['ltpd']=0\nfdf['ltpr']=0\n\noutlierFilter=(fdf['outlierScore']>0)\nprint(outlierFilter.sum())\n\nhipdFilter = (fdf['hmmax']>fdf['hemax']) & (fdf['hmmax']>fdf['hlmax']) & outlierFilter\nhtpdFilter = (fdf['hemax']>fdf['hmmax']) & (fdf['hemax']>fdf['hlmax']) & outlierFilter\nlipdFilter = (fdf['lmmax']>fdf['lemax']) & (fdf['lmmax']>fdf['llmax']) & outlierFilter\nltpdFilter = (fdf['lemax']>fdf['lmmax']) & (fdf['lemax']>fdf['llmax']) & outlierFilter\n\nprint(hipdFilter.sum())\nprint(htpdFilter.sum())\nprint(lipdFilter.sum())\nprint(ltpdFilter.sum())\n\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"3a2706cf1e038c2c4e0203fa655bbeca35a0b657"},"cell_type":"markdown","source":"## Use the filters to set decay values without ifs and loops"},{"metadata":{"trusted":true,"_uuid":"6ac53372722587e5e6539843fc2943e228924dfa"},"cell_type":"code","source":"#peak to peak\n#these are light curves where the peak was in the middle\nfdf.loc[hipdFilter,'hipd']=(fdf.loc[hipdFilter,'hmmax']-fdf.loc[hipdFilter,'hlmax']) / \\\n (fdf.loc[hipdFilter,'hmmxd']-fdf.loc[hipdFilter,'hlmxd'])\nfdf.loc[lipdFilter,'lipd']=(fdf.loc[lipdFilter,'lmmax']-fdf.loc[lipdFilter,'llmax']) / \\\n (fdf.loc[lipdFilter,'lmmxd']-fdf.loc[lipdFilter,'llmxd'])\n\n#these are light curves where the peak was in the beginning\nfdf.loc[htpdFilter,'hipd']=(fdf.loc[htpdFilter,'hemax']-fdf.loc[htpdFilter,'hmmax']) / \\\n (fdf.loc[htpdFilter,'hemxd']-fdf.loc[htpdFilter,'hmmxd'])\nfdf.loc[ltpdFilter,'lipd']=(fdf.loc[ltpdFilter,'lemax']-fdf.loc[ltpdFilter,'lmmax']) / \\\n (fdf.loc[ltpdFilter,'lemxd']-fdf.loc[ltpdFilter,'lmmxd'])\nfdf.loc[htpdFilter,'htpd']=(fdf.loc[htpdFilter,'hmmax']-fdf.loc[htpdFilter,'hlmax']) / \\\n (fdf.loc[htpdFilter,'hmmxd']-fdf.loc[htpdFilter,'hlmxd'])\nfdf.loc[ltpdFilter,'ltpd']=(fdf.loc[ltpdFilter,'lmmax']-fdf.loc[ltpdFilter,'llmax']) / \\\n (fdf.loc[ltpdFilter,'lmmxd']-fdf.loc[ltpdFilter,'llmxd'])\n\n#print(fdf.loc[lipdFilter,'lipd'])\nfdf[outlierFilter].head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"0a111ebab7713e8aac5a3264949264fd7248512c"},"cell_type":"markdown","source":"## Repeat decay stats for rise\n- Not sure if this will have significance but not ruling it out\n- Many of the rises will have negative slopes - which is counterintuitive at first but is because most outliers are flux spikes so even min to min is a decay\n- If an outlier is actually a trough then the 'decay' could have a positive slope"},{"metadata":{"trusted":true,"_uuid":"7b7e8e4a51a12b3c502c173b6b30c8ac7cc33924"},"cell_type":"code","source":"hiprFilter = (fdf['hmmin']