In this Notebook we will be going through and learning the basics of Regular Expressions(RE or Regex) and then applying them to our dataset
\n\n### The Goal: provide a helpful tool for your preprocessing toolset and help me build my own skill set via presentation\n\n#### One of the applied regular expressions we will work with and that was inspiration for this post is from @ammarnassanalhajali in his Original Notebook: Notebook","metadata":{}},{"cell_type":"markdown","source":"
imports
","metadata":{}},{"cell_type":"code","source":"import os\n#for file management\nimport glob\n#for precise file selection(low verbosity)\nimport time\n#return the time in seconds since the epoch as a floating point number. A good way to watch your training progress quantitatively\nimport regex as re\n#import regular expression matching operations. alows for facilitated use of string comparison. Docs:https://docs.python.org/3/library/re.html\nimport math\n#provides mathematical functions established within the c-standard. gives ceil, factorial, etc. Docs:https://docs.python.org/3/library/math.html\nimport numpy as np\n#linear algebra\nimport pandas as pd\n#succinct array and data handling\n","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-08-23T11:59:20.716214Z","iopub.execute_input":"2021-08-23T11:59:20.716729Z","iopub.status.idle":"2021-08-23T11:59:20.876769Z","shell.execute_reply.started":"2021-08-23T11:59:20.716695Z","shell.execute_reply":"2021-08-23T11:59:20.875681Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
Context
\n \n
Regex can be used in order to scrape a webpage, review your code more quickly, parse files in a loop, check user input, and the list goes on. However, it is a library so there are some things we need to know going forward in order to maximize our time working with them. The rest we should(as always) supplement by rereading the documentation when we are profoundly lost(which can happen at first...and in general)
How to: Lets talk special characters, general knowledge, and give an example for reading email notation
","metadata":{}},{"cell_type":"markdown","source":"> taking a peek at special characters","metadata":{}},{"cell_type":"code","source":"# special characters, a few to keep in mind as we continue(see the regex cheat sheet I linked for far more):\n\n# . the default mode that will catch all characters except for new line\n\n# ^ matches the start of the string\n\n# $ matches the end of the string just before the newline at the end of the string\n\n# * matches the preceding RE to as many repetitions of it as there are \n\n# [] match a set of characters\n\n# \\ escape special characters\n\n# | ex. RE1 | RE2: Match either RE1 or RE2\n\n# \\Z match only at the end of a string\n\n# re.findall() return all non-overlapping matches of pattern in string as list of strings\n\n# ? match 0 or 1 repetitions of RE\n\n#working with ranges concepts:\n\n#[0-5][0-9] will match all the two-digits numbers from 00 to 59\n\n#see docs for the special characters list\n\n#non special chracters that used to be special( or ones you may have assumed were special)\n#'!', '\"', '%', \"'\", ',', '/', ':', ';', '<', '=', '>', '@', and \"`\" are no longer escaped.","metadata":{"execution":{"iopub.status.busy":"2021-08-23T11:57:39.760402Z","iopub.execute_input":"2021-08-23T11:57:39.760722Z","iopub.status.idle":"2021-08-23T11:57:39.764559Z","shell.execute_reply.started":"2021-08-23T11:57:39.760693Z","shell.execute_reply":"2021-08-23T11:57:39.763817Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"> some interesting information. turns out regex plus worried engineers can = a boggled reader. but now we know to watch out for it :)","metadata":{}},{"cell_type":"markdown","source":"one last interesting note on escaped characters and special characters: \n\" you'll typically see everything escaped (e.g. [a-z\\%\\$\\#\\@\\!\\-\\_]) simply because the engineer doesn't know what's \"special\" and what's not... so they \"play it safe\" and obfuscate the expression with loads of excessive backslashes.\" \n-[source](https://stackoverflow.com/questions/9589074/regex-should-hyphens-be-escaped)","metadata":{}},{"cell_type":"markdown","source":"> reading an email from user input","metadata":{}},{"cell_type":"code","source":"#lets take user input and return whether or not they inputted an email\n\nemail_pattern = \"[a-zA-Z0-9]+@[a-zA-Z]+\\.(com|edu|net)\"\n# [a-zA-Z0-9] here we are looking to match a set of characters that can be read as follows:\n'''\nstarting from the beginning of the string match if the string has...\nany characters in the lower case alphabet or a-z\nany chracters in the upper case alphabet or A-Z\nand any digits between 0-9\nthus a-zA-Z0-9 where the hypen(-) indicates a range \n'''\n# +@[a-zA-Z] the plus indicates we break/stop when we reach the subsequent value in this case the @ symbol\n\n#+\\.(com|edu|net)\n'''\nso now we have collected a string and an @ symbol and from there we continue matching with the previous logic until we meet +\\. which means:\n+ stop previous matching at a period(.) the backslash is here because a period is a special character \n\n(com|edu|net) where the () indicates that the contents matched inside are the last ones and the | indicates or\nfrom here we match the next section if it includes com or edu or net and ends the matching function\n'''","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2021-08-23T11:59:23.527195Z","iopub.execute_input":"2021-08-23T11:59:23.527615Z","iopub.status.idle":"2021-08-23T11:59:23.532449Z","shell.execute_reply.started":"2021-08-23T11:59:23.527581Z","shell.execute_reply":"2021-08-23T11:59:23.531585Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"> we set our pattern now lets run our logic (I will leave the output after i typed the crazy@floridaman.net email when prompted in initial run)","metadata":{}},{"cell_type":"code","source":"user_input = input()\nif(re.search(email_pattern, user_input)):\n#we run a conditional that uses the search function from the re module that\n#takes the parameter that is your re patter and casts it to your input and \n#returns what I assume is an object that has a few attributes, the one that\n#we care about here being True or False as to if we have an email present in the parsed input\n\n print(\"email\")\nelse:\n print(\"non-email\")","metadata":{"execution":{"iopub.status.busy":"2021-08-23T11:59:33.027747Z","iopub.execute_input":"2021-08-23T11:59:33.028139Z","iopub.status.idle":"2021-08-23T11:59:38.480012Z","shell.execute_reply.started":"2021-08-23T11:59:33.028104Z","shell.execute_reply":"2021-08-23T11:59:38.478296Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
Heads Up! try ctrl-f and then enter in some regular expressions in order to find digits or another example. its a great time saver and allows you to test your implementation before coding it(note that you need to press the 3 dots in the drop down and then click on the .* in the corner of the menu
","metadata":{}},{"cell_type":"markdown","source":"> lets look at our main directory, choose an example subfile and define the pattern it has in regex notation","metadata":{}},{"cell_type":"code","source":"#stepping up the complexity a bit here. skip to the next example for the sorted function used in a notebook for efficiently loading 3d dicom images\n\n'''\n\nLets create the initial_pattern for the data within our main dir and then change it to remove the hyphen in the usual ../\"Image-1\" subfile\n\n'''\n\n#re.sub(initial_pattern, new_pattern, file_path)\ndir = '../input/rsna-miccai-brain-tumor-radiogenomic-classification'\n\np1img1 = 'train/00000/FLAIR/Image-1.dcm' #p1img1 for patient one image one","metadata":{"execution":{"iopub.status.busy":"2021-08-23T11:59:42.099270Z","iopub.execute_input":"2021-08-23T11:59:42.099702Z","iopub.status.idle":"2021-08-23T11:59:42.104944Z","shell.execute_reply.started":"2021-08-23T11:59:42.099661Z","shell.execute_reply":"2021-08-23T11:59:42.103656Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"> now that we have a target path, we need an appropriate regular expression match in order to modify the contents of the pattern to remove the hyphen later","metadata":{}},{"cell_type":"code","source":"#Solution\ninitial_pattern = r\"(\\w+/\\w+/\\w+/[a-zA-Z0-9]+)(\\-)(\\d\\.dcm)\"\n#we set the initial_pattern variable to a raw string whose contents we make 3 groups, 1 and 3 separate from 2 that contains the hyphen \n#\\w indicates any character\n#\\w+ indicates any word character of an entire string until you stop it from matching anymore at a different, non word string value\n # w+ == [a-zA-Z0-9] , i just use both here because i initially was afraid of doing a w++ while trying to end the w+ statement\n#\\- indicates the hypen\n# () indicates a grouping and it is excluded from what is read as the pattern semantically\n#\\d indicates one digit between 0-9 \n#\\. indicates a period\n#dcm is just the matching to the letters dcm spelled out\n# in effect we simply concatenate the matched values in order with a forward slash(/) when indicated and have also made sure to separate the hyphen so we can remove it in the next code block","metadata":{"execution":{"iopub.status.busy":"2021-08-23T11:59:44.587071Z","iopub.execute_input":"2021-08-23T11:59:44.587788Z","iopub.status.idle":"2021-08-23T11:59:44.593472Z","shell.execute_reply.started":"2021-08-23T11:59:44.587732Z","shell.execute_reply":"2021-08-23T11:59:44.592434Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"> we can quickly run a test with this script and use some conditionals to check and change our path value","metadata":{}},{"cell_type":"code","source":"test_path = 'train/00000/FLAIR/Image-1.dcm'\nif(re.search(initial_pattern, test_path )):\n #the search function compares the initial_pattern and the path and determines if they have the same regex representation\n print(\"works\")\n new = re.sub(initial_pattern, r'\\1\\3', test_path) #exclude grouping 2, \\2 that holds the hyphen\n #the re.sub() method substitutes the values of our initial pattern out using the middle parameter which selects which groups we would like to read from our test_path that is casted with the initial_pattern\n print(new)\nelse: #as usual, regex involves some guess and check so a conditional with some level of feedback is very helpful\n print(\"nah\")\n# initial_pattern = re.compile(initial_pattern)\n print(initial_pattern)\n #keep","metadata":{"execution":{"iopub.status.busy":"2021-08-23T11:59:47.407922Z","iopub.execute_input":"2021-08-23T11:59:47.408654Z","iopub.status.idle":"2021-08-23T11:59:47.420146Z","shell.execute_reply.started":"2021-08-23T11:59:47.408589Z","shell.execute_reply":"2021-08-23T11:59:47.418803Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"> and as we had guessed earlier there is indeed an object that logs the info about an re","metadata":{}},{"cell_type":"code","source":"# initial_pattern = re.compile(initial_pattern)\nprint(re.search(initial_pattern, test_path))","metadata":{"execution":{"iopub.status.busy":"2021-08-23T11:59:51.234494Z","iopub.execute_input":"2021-08-23T11:59:51.234953Z","iopub.status.idle":"2021-08-23T11:59:51.240371Z","shell.execute_reply.started":"2021-08-23T11:59:51.234902Z","shell.execute_reply":"2021-08-23T11:59:51.239585Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
Applied: Sorted function for
","metadata":{}},{"cell_type":"code","source":"#for now we will ignore the context and just plug in some params so we can test the logic\n\nscan_id = '00001'\ndata_directory = '../input/rsna-miccai-brain-tumor-radiogenomic-classification/train/00000/FLAIR/Image-1.dcm' \n\ndef load_dicom_images_3d(scan_id, num_imgs=64, img_size=256, mri_type=\"FLAIR\", split=\"train\", rotate=0):\n\n files = sorted(glob.glob(f\"{data_directory}/{split}/{scan_id}/{mri_type}/*.dcm\"), \n #we set files equal to a sorted function\n #glob.glob function parses an f string using our input parameters\n \n key=lambda var:[int(x) if x.isdigit() else x for x in re.findall(r'[^0-9]|[0-9]+', var)])\n #we set our key value that will dictate our sorting to a lambda function that utilizes an regexpression\n #logic annotated in expanded form example below\n \n #lambda function in expanded form\n '''\n --define lambda function--\n def lambda(var):\n ---for x in the contents of the 1d array generated by parsing each individual character grouping via the regex expression and re.findall expression---\n for x in re.findall((r'[^0-9]|[0-9]+', var)):\n ---if x is a digit, return the value as an integer in an array and using the sorted value use that mapped array that was generated at the given-----\n ---values to return a files variable the is in numerical ascending order based on Image-#--------\n if x.isdigit():\n return int(x)\n else:\n return x\n \n ''' ","metadata":{"execution":{"iopub.status.busy":"2021-08-23T11:57:48.994354Z","iopub.status.idle":"2021-08-23T11:57:48.994797Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"> what the re.findall is doing in the background","metadata":{}},{"cell_type":"code","source":"file = sorted(glob.glob('../input/rsna-miccai-brain-tumor-radiogenomic-classification/train/00000/FLAIR/Image-1.dcm'), \n #we set files equal to a sorted function\n #glob.glob function parses an f string using our input parameters\n\n key=lambda var:[int(x) if x.isdigit() else x for x in re.findall(r'[^0-9]|[0-9]+', var)])\nprint(file)","metadata":{"execution":{"iopub.status.busy":"2021-08-23T11:57:48.995673Z","iopub.status.idle":"2021-08-23T11:57:48.996068Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pattern = r'[^0-9]|[0-9]+'\nre.findall(pattern, '../input/rsna-miccai-brain-tumor-radiogenomic-classification/train/00000/FLAIR/Image-1.dcm' )","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2021-08-23T12:00:03.500279Z","iopub.execute_input":"2021-08-23T12:00:03.500703Z","iopub.status.idle":"2021-08-23T12:00:03.510622Z","shell.execute_reply.started":"2021-08-23T12:00:03.500666Z","shell.execute_reply":"2021-08-23T12:00:03.508821Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
Applied Bonus=>identify files from within directories and access them via a loop
\n\n> source: https://stackoverflow.com/questions/39293968/how-do-i-search-directories-and-find-files-that-match-regex","metadata":{}},{"cell_type":"code","source":"import os\n#importing os to enable ability to change path values from a script\nimport re\n#importing regular expressions (import regex as re can help prevent errors at times but we'll leave this import as it was from its post)\n\nregex = re.compile('(.*zip$)|(.*rar$)|(.*r01$)')\n#create a regex object using the re.compile function that is a grouping of either a match to a zip file, rar file, or ro1 file\n # .* effectively grabs all the characters of a raw string with the . and then for as many iterations that there are but is then capped at the end with the word zip as it is connected to a $ end of string indicator\n #this works so well because we know nothing comes after the file type in a file name\nrx = '(.*zip$)|(.*rar$)|(.*r01$)'\n#set a string to be 3 possible groupings\n\nfor root, dirs, files in os.walk(\"../Documents\"):\n #traverse through the builtin root dirs and files of frame of the os library with the walk function at the given folder \n for file in files:\n #for files in files use the re.match regex function to check if at any group 1,2,or3 that we setup in rx = 1|2|3 we return\n #a print statement declaring what file type it is. from there you can make functionalities based on the file type\n res = re.match(rx, file)\n if res:\n if res.group(1):\n print(\"ZIP\",file)\n if res.group(2):\n print(\"RAR\",file)\n if res.group(3):\n print(\"R01\",file)","metadata":{"execution":{"iopub.status.busy":"2021-08-23T11:57:48.998884Z","iopub.status.idle":"2021-08-23T11:57:48.999284Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"for those who made it to here, here is a funny stackoverflow post that I found on my quest to comprehend regex: https://stackoverflow.com/questions/1454913/regular-expression-to-find-a-string-included-between-two-characters-while-exclud\n\n\"Easy done\"-Cletus ","metadata":{}},{"cell_type":"markdown","source":"hop in our ml discord! \n\n@ https://discord.gg/E5hJkwNT","metadata":{}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}