{ "id": 290235, "title": "A cross-validation strategy: subsequences", "url": "/competitions/tensorflow-great-barrier-reef/discussion/290235", "author_name": "", "post_date": "2021-11-23T17:15:43.910357400Z", "votes": 37, "comment_count": 11, "views": 0, "content": "

Hello community,

\n

You have probably already discovered that there are 3 videos and that that makes cross-validation a bit weird, since this means a 3-fold CV or a weird 66-33 split.

\n

Sequences, on the other hand, are not much better: there are only 20 and they are quite dissimilar in their sizes.

\n

I have come up with a \"subsequence\" concept, which creates 137 \"splitting units\".

\n

A subsequence is a piece of a sequence when objects are continually present.
\nOr the opposite, a continuous piece when objects are not present.

\n

 

\n

Let's see an example. Consider the sequence A with the following frames:

\n\n

In this case, we say that the sequence A has 4 subsequences (1-20, 21-30, 31-60, 61-80).

\n

 

\n

A subsequence seems to me like the minimal atom for ensuring no leaks happen between train and test.

\n

I have gone in-depth with this approach and created common splits with it in this notebook:

\n

🐠 Reef - CV strategy: subsequences!

\n

The resulting dataframes are saved in this dataset:

\n

reef-cv-strategy-subsequences-dataframes

\n

There are train-validation splits with 1%, 5%, 10% and 20% validation sizes and 5 and 10 stratified folds for cross-validation that are ready to use.

\n

I'm planning to adapt my notebooks to work with these splits soon. Hopefully you will find them useful.

\n

On the other hand: have you thought of other approaches? I could think of this one only, but I imagine others are possible as well and I would love to hear about them.

", "messages": [ { "id": "1593146", "postDate": "11/23/2021 17:15:43", "content": "

Hello community,

\n

You have probably already discovered that there are 3 videos and that that makes cross-validation a bit weird, since this means a 3-fold CV or a weird 66-33 split.

\n

Sequences, on the other hand, are not much better: there are only 20 and they are quite dissimilar in their sizes.

\n

I have come up with a \"subsequence\" concept, which creates 137 \"splitting units\".

\n

A subsequence is a piece of a sequence when objects are continually present.
\nOr the opposite, a continuous piece when objects are not present.

\n

 

\n

Let's see an example. Consider the sequence A with the following frames:

\n\n

In this case, we say that the sequence A has 4 subsequences (1-20, 21-30, 31-60, 61-80).

\n

 

\n

A subsequence seems to me like the minimal atom for ensuring no leaks happen between train and test.

\n

I have gone in-depth with this approach and created common splits with it in this notebook:

\n

🐠 Reef - CV strategy: subsequences!

\n

The resulting dataframes are saved in this dataset:

\n

reef-cv-strategy-subsequences-dataframes

\n

There are train-validation splits with 1%, 5%, 10% and 20% validation sizes and 5 and 10 stratified folds for cross-validation that are ready to use.

\n

I'm planning to adapt my notebooks to work with these splits soon. Hopefully you will find them useful.

\n

On the other hand: have you thought of other approaches? I could think of this one only, but I imagine others are possible as well and I would love to hear about them.

", "rawMarkdown": "Hello community,\n\nYou have probably already discovered that there are 3 videos and that that makes cross-validation a bit weird, since this means a 3-fold CV or a weird 66-33 split.\n\n**Sequences**, on the other hand, are not much better: there are only 20 and they are quite dissimilar in their sizes.\n\nI have come up with a \"subsequence\" concept, which creates 137 \"splitting units\".\n\nA **subsequence** is a piece of a sequence when objects are continually present.\nOr the opposite, a continuous piece when objects are not present.\n\n \n\nLet's see an example. Consider the sequence `A` with the following frames:\n* `1-20` - No annotations present\n* `21-30` - Annotations present\n* `31-60` - No annotations\n* `61-80` - Annotations present\n\nIn this case, we say that the sequence `A` has `4` subsequences (`1-20`, `21-30`, `31-60`, `61-80`).\n\n \n\n\nA subsequence seems to me like the minimal atom for ensuring no leaks happen between train and test.\n\nI have gone in-depth with this approach and created common splits with it in **this notebook**:\n### [🐠 Reef - CV strategy: subsequences!](https://www.kaggle.com/julian3833/reef-cv-strategy-subsequences)\n\nThe resulting dataframes are saved in **this dataset**: \n### [reef-cv-strategy-subsequences-dataframes](https://www.kaggle.com/julian3833/reef-cv-strategy-subsequences-dataframes/)\n\n\n#### There are train-validation splits with 1%, 5%, 10% and 20% validation sizes and 5 and 10 stratified folds for cross-validation that are ready to use.\n\nI'm planning to adapt my notebooks to work with these splits soon. Hopefully you will find them useful.\n\n### On the other hand: have you thought of other approaches? I could think of this one only, but I imagine others are possible as well and I would love to hear about them.", "votes": null }, { "id": "1593821", "postDate": "11/24/2021 09:51:59", "content": "

Good approach. but what about the edges of that sequence?
\nLet say you have a video and divide it into sequences [1-20, 21-40, 41-60, 61-80, 81-100] or [A, B, C, D, E, F]. So, you will feed A, C, E into the training process, and B, D, F for validation, right?

\n

But what about the similarity between A[19] and B[0]? Subsequence frames will be almost the same. Will overfitted model predict the head-tail of the validation sequence correctly?

", "rawMarkdown": "Good approach. but what about the edges of that sequence?\nLet say you have a video and divide it into sequences `[1-20, 21-40, 41-60, 61-80, 81-100]` or `[A, B, C, D, E, F]`. So, you will feed A, C, E into the training process, and B, D, F for validation, right?\n\nBut what about the similarity between A[19] and B[0]? Subsequence frames will be almost the same. Will overfitted model predict the head-tail of the validation sequence correctly?", "votes": null }, { "id": "1594453", "postDate": "11/24/2021 21:03:15", "content": "

Hi Mykola, thanks for commenting.

\n

The edges are kind of covered by the fact that two consecutive subsequences cannot have both objects. So if A has objects in it, then B doesn't. Only C will have objects, but they will appear in a region far from A.

\n

Anyway, A[19] and B[0] are continuous frames as you correctly stated, so it is still might be somehow problematic. I think this problem is minor though, but it's just an intuition. Putting continuous subsequences in the same bag (aka training or validation) can strongly reduce these edge cases, but in that case, intra-training and intra-validation variance will be reduced as well (the subsequence ids in the data frames can be used to simply create different splits based on them).

", "rawMarkdown": "Hi Mykola, thanks for commenting.\n\nThe edges are kind of covered by the fact that two consecutive subsequences cannot have both objects. So if A has objects in it, then B doesn't. Only C will have objects, but they will appear in a region far from A. \n\nAnyway, A[19] and B[0] are continuous frames as you correctly stated, so it is still might be somehow problematic. I think this problem is minor though, but it's just an intuition. Putting continuous subsequences in the same bag (aka training or validation) can strongly reduce these edge cases, but in that case, intra-training and intra-validation variance will be reduced as well (the subsequence ids in the data frames can be used to simply create different splits based on them).", "votes": null }, { "id": "1660076", "postDate": "01/22/2022 11:30:38", "content": "

I just come up with that sequence type cross validation, and discover that you have already published it! Thanks bro.

", "rawMarkdown": "I just come up with that sequence type cross validation, and discover that you have already published it! Thanks bro.", "votes": null }, { "id": "1669904", "postDate": "01/31/2022 03:54:46", "content": "

Great work. Thanks for sharing, this is helpful. I'm using your sub sequences.

\n

Have you considered sklearn.model_selection.StratifiedGroupKFold?

\n
from sklearn.model_selection import StratifiedGroupKFold\nsgkf = StratifiedGroupKFold(n_splits=5)\nfor fold, (t_idx, v_idx) in enumerate( sgkf.split(df, df.has_annotations, df.subsequence_id) ):\n    df.loc[v_idx,'fold'] = fold\n
\n

This produces 5 folds where each fold has 4700 frames and each fold has 984 annotated frames.

", "rawMarkdown": "Great work. Thanks for sharing, this is helpful. I'm using your sub sequences. \n\nHave you considered `sklearn.model_selection.StratifiedGroupKFold`? \n\n from sklearn.model_selection import StratifiedGroupKFold\n sgkf = StratifiedGroupKFold(n_splits=5)\n for fold, (t_idx, v_idx) in enumerate( sgkf.split(df, df.has_annotations, df.subsequence_id) ):\n df.loc[v_idx,'fold'] = fold\n\nThis produces 5 folds where each fold has 4700 frames and each fold has 984 annotated frames.", "votes": null }, { "id": "1671971", "postDate": "02/01/2022 20:11:52", "content": "

Nice code! For those who want to use this code on Kaggle - you'll have to update scikit-learn package. You can do it with this code:

\n
!pip uninstall -q -y scikit-learn\n!pip install -q scikit-learn\n
", "rawMarkdown": "Nice code! For those who want to use this code on Kaggle - you'll have to update scikit-learn package. You can do it with this code:\n```\n!pip uninstall -q -y scikit-learn\n!pip install -q scikit-learn\n```", "votes": null }, { "id": "1672579", "postDate": "02/02/2022 07:09:16", "content": "

@cdeotte what is the grouping of cv strategy? by sequence?

", "rawMarkdown": "cdeotte what is the grouping of cv strategy? by sequence?", "votes": null }, { "id": "1673103", "postDate": "02/02/2022 14:12:36", "content": "

No, group by sub-sequence which are made in notebook that is linked above.

", "rawMarkdown": "No, group by `sub-sequence` which are made in notebook that is linked above.", "votes": null }, { "id": "1673187", "postDate": "02/02/2022 15:06:07", "content": "

@cdeotte , thanks, I overlooked \"df\" :D

", "rawMarkdown": "cdeotte , thanks, I overlooked \"df\" :D", "votes": null }, { "id": "1673938", "postDate": "02/03/2022 05:29:19", "content": "

Great work. Thanks for sharing, this is helpful.

", "rawMarkdown": "Great work. Thanks for sharing, this is helpful.", "votes": null }, { "id": "1676122", "postDate": "02/04/2022 17:34:35", "content": "

Why the F2 reporting hits high 0.7 to 0.8 range with this subsequence strategy ? I'm using the 0.1 train validation split

", "rawMarkdown": "Why the F2 reporting hits high 0.7 to 0.8 range with this subsequence strategy ? I'm using the 0.1 train validation split", "votes": null }, { "id": "1676130", "postDate": "02/04/2022 17:44:13", "content": "

I think it is because when Yolo reports F2, it is computing F2 per image (then average). Whereas Kaggle competition computes F2 per everything at once.

\n

(I have also have a difference between what Yolo reports and what we get when computing competition metric ourselves).

", "rawMarkdown": "I think it is because when Yolo reports F2, it is computing F2 per image (then average). Whereas Kaggle competition computes F2 per everything at once.\n\n(I have also have a difference between what Yolo reports and what we get when computing competition metric ourselves).", "votes": null } ], "comments": [ { "id": 1593821, "author_name": "meowmeowmeowmeowmeow", "author_url": "", "post_date": "11/24/2021 09:51:59", "content": "

Good approach. but what about the edges of that sequence?
\nLet say you have a video and divide it into sequences [1-20, 21-40, 41-60, 61-80, 81-100] or [A, B, C, D, E, F]. So, you will feed A, C, E into the training process, and B, D, F for validation, right?

\n

But what about the similarity between A[19] and B[0]? Subsequence frames will be almost the same. Will overfitted model predict the head-tail of the validation sequence correctly?

", "votes": null, "replies": [ { "id": 1594453, "author_name": "julian3833", "author_url": "", "post_date": "11/24/2021 21:03:15", "content": "

Hi Mykola, thanks for commenting.

\n

The edges are kind of covered by the fact that two consecutive subsequences cannot have both objects. So if A has objects in it, then B doesn't. Only C will have objects, but they will appear in a region far from A.

\n

Anyway, A[19] and B[0] are continuous frames as you correctly stated, so it is still might be somehow problematic. I think this problem is minor though, but it's just an intuition. Putting continuous subsequences in the same bag (aka training or validation) can strongly reduce these edge cases, but in that case, intra-training and intra-validation variance will be reduced as well (the subsequence ids in the data frames can be used to simply create different splits based on them).

", "votes": null, "replies": [] } ] }, { "id": 1660076, "author_name": "calvchen", "author_url": "", "post_date": "01/22/2022 11:30:38", "content": "

I just come up with that sequence type cross validation, and discover that you have already published it! Thanks bro.

", "votes": null, "replies": [] }, { "id": 1669904, "author_name": "cdeotte", "author_url": "", "post_date": "01/31/2022 03:54:46", "content": "

Great work. Thanks for sharing, this is helpful. I'm using your sub sequences.

\n

Have you considered sklearn.model_selection.StratifiedGroupKFold?

\n
from sklearn.model_selection import StratifiedGroupKFold\nsgkf = StratifiedGroupKFold(n_splits=5)\nfor fold, (t_idx, v_idx) in enumerate( sgkf.split(df, df.has_annotations, df.subsequence_id) ):\n    df.loc[v_idx,'fold'] = fold\n
\n

This produces 5 folds where each fold has 4700 frames and each fold has 984 annotated frames.

", "votes": null, "replies": [ { "id": 1671971, "author_name": "atamazian", "author_url": "", "post_date": "02/01/2022 20:11:52", "content": "

Nice code! For those who want to use this code on Kaggle - you'll have to update scikit-learn package. You can do it with this code:

\n
!pip uninstall -q -y scikit-learn\n!pip install -q scikit-learn\n
", "votes": null, "replies": [] }, { "id": 1672579, "author_name": "projdev", "author_url": "", "post_date": "02/02/2022 07:09:16", "content": "

@cdeotte what is the grouping of cv strategy? by sequence?

", "votes": null, "replies": [] }, { "id": 1673103, "author_name": "cdeotte", "author_url": "", "post_date": "02/02/2022 14:12:36", "content": "

No, group by sub-sequence which are made in notebook that is linked above.

", "votes": null, "replies": [] }, { "id": 1673187, "author_name": "projdev", "author_url": "", "post_date": "02/02/2022 15:06:07", "content": "

@cdeotte , thanks, I overlooked \"df\" :D

", "votes": null, "replies": [] }, { "id": 1676122, "author_name": "nyleve", "author_url": "", "post_date": "02/04/2022 17:34:35", "content": "

Why the F2 reporting hits high 0.7 to 0.8 range with this subsequence strategy ? I'm using the 0.1 train validation split

", "votes": null, "replies": [] }, { "id": 1676130, "author_name": "cdeotte", "author_url": "", "post_date": "02/04/2022 17:44:13", "content": "

I think it is because when Yolo reports F2, it is computing F2 per image (then average). Whereas Kaggle competition computes F2 per everything at once.

\n

(I have also have a difference between what Yolo reports and what we get when computing competition metric ourselves).

", "votes": null, "replies": [] } ] }, { "id": 1673938, "author_name": "spectreprediction", "author_url": "", "post_date": "02/03/2022 05:29:19", "content": "

Great work. Thanks for sharing, this is helpful.

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1593146": "Hello community,\n\nYou have probably already discovered that there are 3 videos and that that makes cross-validation a bit weird, since this means a 3-fold CV or a weird 66-33 split.\n\n**Sequences**, on the other hand, are not much better: there are only 20 and they are quite dissimilar in their sizes.\n\nI have come up with a \"subsequence\" concept, which creates 137 \"splitting units\".\n\nA **subsequence** is a piece of a sequence when objects are continually present.\nOr the opposite, a continuous piece when objects are not present.\n\n \n\nLet's see an example. Consider the sequence `A` with the following frames:\n* `1-20` - No annotations present\n* `21-30` - Annotations present\n* `31-60` - No annotations\n* `61-80` - Annotations present\n\nIn this case, we say that the sequence `A` has `4` subsequences (`1-20`, `21-30`, `31-60`, `61-80`).\n\n \n\n\nA subsequence seems to me like the minimal atom for ensuring no leaks happen between train and test.\n\nI have gone in-depth with this approach and created common splits with it in **this notebook**:\n### [🐠 Reef - CV strategy: subsequences!](https://www.kaggle.com/julian3833/reef-cv-strategy-subsequences)\n\nThe resulting dataframes are saved in **this dataset**: \n### [reef-cv-strategy-subsequences-dataframes](https://www.kaggle.com/julian3833/reef-cv-strategy-subsequences-dataframes/)\n\n\n#### There are train-validation splits with 1%, 5%, 10% and 20% validation sizes and 5 and 10 stratified folds for cross-validation that are ready to use.\n\nI'm planning to adapt my notebooks to work with these splits soon. Hopefully you will find them useful.\n\n### On the other hand: have you thought of other approaches? I could think of this one only, but I imagine others are possible as well and I would love to hear about them.", "1593821": "Good approach. but what about the edges of that sequence?\nLet say you have a video and divide it into sequences `[1-20, 21-40, 41-60, 61-80, 81-100]` or `[A, B, C, D, E, F]`. So, you will feed A, C, E into the training process, and B, D, F for validation, right?\n\nBut what about the similarity between A[19] and B[0]? Subsequence frames will be almost the same. Will overfitted model predict the head-tail of the validation sequence correctly?", "1594453": "Hi Mykola, thanks for commenting.\n\nThe edges are kind of covered by the fact that two consecutive subsequences cannot have both objects. So if A has objects in it, then B doesn't. Only C will have objects, but they will appear in a region far from A. \n\nAnyway, A[19] and B[0] are continuous frames as you correctly stated, so it is still might be somehow problematic. I think this problem is minor though, but it's just an intuition. Putting continuous subsequences in the same bag (aka training or validation) can strongly reduce these edge cases, but in that case, intra-training and intra-validation variance will be reduced as well (the subsequence ids in the data frames can be used to simply create different splits based on them).", "1660076": "I just come up with that sequence type cross validation, and discover that you have already published it! Thanks bro.", "1669904": "Great work. Thanks for sharing, this is helpful. I'm using your sub sequences. \n\nHave you considered `sklearn.model_selection.StratifiedGroupKFold`? \n\n from sklearn.model_selection import StratifiedGroupKFold\n sgkf = StratifiedGroupKFold(n_splits=5)\n for fold, (t_idx, v_idx) in enumerate( sgkf.split(df, df.has_annotations, df.subsequence_id) ):\n df.loc[v_idx,'fold'] = fold\n\nThis produces 5 folds where each fold has 4700 frames and each fold has 984 annotated frames.", "1671971": "Nice code! For those who want to use this code on Kaggle - you'll have to update scikit-learn package. You can do it with this code:\n```\n!pip uninstall -q -y scikit-learn\n!pip install -q scikit-learn\n```", "1672579": "cdeotte what is the grouping of cv strategy? by sequence?", "1673103": "No, group by `sub-sequence` which are made in notebook that is linked above.", "1673187": "cdeotte , thanks, I overlooked \"df\" :D", "1673938": "Great work. Thanks for sharing, this is helpful.", "1676122": "Why the F2 reporting hits high 0.7 to 0.8 range with this subsequence strategy ? I'm using the 0.1 train validation split", "1676130": "I think it is because when Yolo reports F2, it is computing F2 per image (then average). Whereas Kaggle competition computes F2 per everything at once.\n\n(I have also have a difference between what Yolo reports and what we get when computing competition metric ourselves)." }, "source": "meta" }