[BUG] Tracking issue for known flaky tests
#415 opened on Dec 9, 2022
Repository metrics
- Stars
- (31 stars)
- PR merge metrics
- (PR metrics pending)
Description
Please add more flaky tests to this issue as they're discovered; or notes from investigations into the causes of the flaky tests. Reducing flakiness will increase confidence in our automated testing, and reduce the amount of time needed to merge changes as rerunning cypress tests is a time-consuming process.
List of known flaky tests
bucket_level_monitor_spec.js>Bucket-Level Monitors>can be created>by visual editor- This test case seems to be much less flaky when executed locally, but will occasionally fail when executed by github actions. The following error is observed when the test fails.
CypressError: Timed out retrying after 10000ms: `cy.select()` failed because it could not find a single `<option>` with value or text matching: `avg_products_base_price`
query_level_monitor_spec.js>schedule component displays as intended>for a cron expression scheduleThis test case seems to be much less flaky when executed locally, but will occasionally fail when executed by github actions. The following error is observed when the test fails.This flaky behavior is also rarely seen impacting thequery_level_monitor_spec.js>schedule component displays as intended>for an interval scheduletest.
AssertionError: Timed out retrying after 20000ms: Expected to find element: `[data-test-subj="frequency_field"]`, but never found it.
at Context.eval (https://search-t-nobody-avjv-e2ealerting-3rb2m37iqureafmtutcxcgypem.eu-west-1.es-staging.amazonaws.com/__cypress/tests?p=cypress/integration/plugins/alerting-dashboards-plugin/query_level_monitor_spec.js:580:10)
bucket_level_monitor_spec.js>Bucket-Level Monitors>can be created>by visual editorseems to be flaky when executed by the 1.3 functional test repo branch. It has failed with the error below.This seems to relate to the use of{enter}when usingcy.type()here. Cypress fails to execute the{enter}press.The screenshot below helps illustrate this; theenterkey press doesn't execute, so the UI doesn't detect that aTime Fieldhas been defined which prevents the subsequent UI from rendering.A potential solution may involve changing a.blur()call at the end of.type()calls that use{enter}.
AssertionError: Timed out retrying: Expected to find element: `[data-test-subj="addMetricButton"]`, but never found it.
-
cluster_metrics_monitor_spec.js>can be created>for the Cluster Health APIseems to occasionally fail when the API is selected from the dropdown before page finishes loading. This causes the selection to reset which causes the test to fail as it expects to see UI elements for the selection. See image screenshot below -
monitors_dashboard_spec.js>Displays expected number of alertsoccasionally fails with the following error (example). May be related to a delay in alert ingestion.
AssertionError: Timed out retrying after 20000ms: Expected to find element: `:contains(monitors_dashboard_cypress_query_level)`, but never found it. Queried from element: <tr.euiTableRow>
composite_level_monitor_spec.js>can be edited>by visual editoroccasionally fails with the following error (example). Adding a short wait after thiscy.visitcall (link) helps to alleviate the flakiness; but a more permanent solution should be implemented.
1) CompositeLevelMonitor
can be edited
by visual editor:
AssertionError: Timed out retrying after 10000ms: Expected to find element: `[data-test-subj="monitors_list_2"]`, but never found it.
at Context.eval (http://localhost:5601/__cypress/tests?p=cypress/integration/composite_level_monitor_spec.js:339:10)