Create Task
Maniphest T255855

PHP Warning: [data-update-failed]: A data update callback triggered an exception (Fail-safe exception. Avoiding infinite loop due to possibily undetectable existing records in master. It may be due to encoding incompatibility between database values and values passed in $neededRecords parameter.) [Called from Wikibase\Repo\Content\DataUpdateAdapter::doUpdate in /srv/mediawiki/php-1.35.0-wmf.37/extensions/Wikibase/repo/includes/Content/DataUpdateAdapter.php at line 62]
Closed, ResolvedPublicPRODUCTION ERROR
Actions

Description

Error

MediaWiki version: 1.35.0-wmf.37

message
PHP Warning: [data-update-failed]: A data update callback triggered an exception (Fail-safe exception. Avoiding infinite loop due to possibily undetectable existing records in master.
 It may be due to encoding incompatibility between database values and values passed in $neededRecords parameter.) [Called from Wikibase\Repo\Content\DataUpdateAdapter::doUpdate in /srv/mediawiki/php-1.35.0-wmf.37/extensions/Wikibase/repo/includes/Content/DataUpdateAdapter.php at line 62]

Impact

  • Seems to have started today at around 8:20 UTC. Looks like 100 events per 30 minutes.

Notes

Details

Request ID
140f3eda-6e6f-4223-bcc6-d7786bb293f4
Request URL
https://www.wikidata.org/w/api.php
Stack Trace
exception.trace
#0 [internal function]: MWExceptionHandler::handleError(integer, string, string, integer, array)
#1 /srv/mediawiki/php-1.35.0-wmf.37/includes/debug/MWDebug.php(393): trigger_error(string, integer)
#2 /srv/mediawiki/php-1.35.0-wmf.37/includes/debug/MWDebug.php(196): MWDebug::sendMessage(string, string, integer)
#3 /srv/mediawiki/php-1.35.0-wmf.37/includes/GlobalFunctions.php(1064): MWDebug::warning(string, integer, integer, string)
#4 /srv/mediawiki/php-1.35.0-wmf.37/extensions/Wikibase/lib/includes/Reporting/LogWarningExceptionHandler.php(28): wfLogWarning(string, integer)
#5 /srv/mediawiki/php-1.35.0-wmf.37/extensions/Wikibase/repo/includes/Content/DataUpdateAdapter.php(62): Wikibase\Lib\Reporting\LogWarningExceptionHandler->handleException(Exception, string, string)
#6 /srv/mediawiki/php-1.35.0-wmf.37/includes/deferred/DeferredUpdates.php(466): Wikibase\Repo\Content\DataUpdateAdapter->doUpdate()
#7 /srv/mediawiki/php-1.35.0-wmf.37/includes/deferred/RefreshSecondaryDataUpdate.php(101): DeferredUpdates::attemptUpdate(Wikibase\Repo\Content\DataUpdateAdapter, Wikimedia\Rdbms\LBFactoryMulti)
#8 /srv/mediawiki/php-1.35.0-wmf.37/includes/deferred/DeferredUpdates.php(466): RefreshSecondaryDataUpdate->doUpdate()
#9 /srv/mediawiki/php-1.35.0-wmf.37/includes/deferred/DeferredUpdates.php(343): DeferredUpdates::attemptUpdate(RefreshSecondaryDataUpdate, Wikimedia\Rdbms\LBFactoryMulti)
#10 /srv/mediawiki/php-1.35.0-wmf.37/includes/deferred/DeferredUpdates.php(278): DeferredUpdates::run(RefreshSecondaryDataUpdate, Wikimedia\Rdbms\LBFactoryMulti, Monolog\Logger, BufferingStatsdDataFactory, string)
#11 /srv/mediawiki/php-1.35.0-wmf.37/includes/deferred/DeferredUpdates.php(194): DeferredUpdates::handleUpdateQueue(array, string, integer)
#12 /srv/mediawiki/php-1.35.0-wmf.37/includes/MediaWiki.php(1072): DeferredUpdates::doUpdates(string)
#13 /srv/mediawiki/php-1.35.0-wmf.37/includes/MediaWiki.php(849): MediaWiki->restInPeace()
#14 /srv/mediawiki/php-1.35.0-wmf.37/includes/MediaWiki.php(861): MediaWiki->{closure}()
#15 /srv/mediawiki/php-1.35.0-wmf.37/api.php(119): MediaWiki->doPostOutputShutdown()
#16 /srv/mediawiki/w/api.php(3): require(string)
#17 {main}
SubjectRepoBranchLines +/-
More logging for ReplicaMasterAwareRecordIdsAcquirer error casemediawiki/extensions/Wikibasemaster+11 -1
AdHocLogging for ReplicaMasterAwareRecordIdsAcquirermediawiki/extensions/Wikibasewmf/1.35.0-wmf.37+10 -0
Customize query in gerrit

Related Objects

Event Timeline

Michael created this task.Jun 19 2020, 10:27 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 19 2020, 10:27 AM
Michael updated the task description. (Show Details)Jun 19 2020, 10:28 AM
Michael added a subscriber: Lucas_Werkmeister_WMDE.Jun 19 2020, 10:32 AM
Lucas_Werkmeister_WMDE updated the task description. (Show Details)Jun 19 2020, 10:36 AM
Michael triaged this task as Unbreak Now! priority.Jun 19 2020, 10:41 AM
Restricted Application added a subscriber: Liuxinyu970226. · View Herald TranscriptJun 19 2020, 10:41 AM
Michael updated the task description. (Show Details)Jun 19 2020, 10:42 AM
Ladsgroup added a comment.Jun 19 2020, 11:34 AM

Possible duplicate of T252061: [data-update-failed]: Avoiding infinite loop due to possibily undetectable existing records (Wikibase DataUpdateAdapter)

gerritbot added a comment.Jun 19 2020, 1:24 PM

Change 606691 had a related patch set uploaded (by Addshore; owner: Addshore):
[mediawiki/extensions/Wikibase@master] AdHocLogging for ReplicaMasterAwareRecordIdsAcquirer

https://gerrit.wikimedia.org/r/606691

gerritbot added a project: Patch-For-Review.Jun 19 2020, 1:24 PM

Change 606692 had a related patch set uploaded (by Addshore; owner: Addshore):
[mediawiki/extensions/Wikibase@wmf/1.35.0-wmf.37] AdHocLogging for ReplicaMasterAwareRecordIdsAcquirer

https://gerrit.wikimedia.org/r/606692

Michael lowered the priority of this task from Unbreak Now! to High.Jun 19 2020, 1:40 PM

After debugging together with @Addshore, I'm lowering the priority for now as it doesn't seem to do permanent damage. It seems to go wrong somewhere in ReplicaMasterAwareRecordIdsAcquirer::insertNonExistingRecords, but we don't know where/how exactly.

We have a patch that adds more debugging output (see above), but we decided against deploying it on a Friday.

Michael moved this task from Incoming to Wikidata-Campsite-Iteration-∞ (On Hold) on the Wikidata-Campsite board.Jun 19 2020, 1:48 PM
Michael edited projects, added Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)); removed Wikidata-Campsite.
Michael moved this task from To Do (prioritised from top to bottom) to Doing on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.
Maintenance_bot moved this task from incoming to in progress on the Wikidata board.Jun 19 2020, 2:15 PM
Addshore added a comment.Jun 20 2020, 9:51 AM

Just noting here that this didn't stop yet.

image.png (314×1 px, 37 KB)

Michael added a comment.Jun 22 2020, 8:49 AM

We are currently speculating that these errors might be triggered by a batch of quickstatement edits adding labels to Chinese villages.

Two candidates are:

Michael added a comment.Jun 22 2020, 8:51 AM

The errors reduced their frequency abruptly over the weekend, but are occasionally still happening:

image.png (189×1 px, 23 KB)

Addshore claimed this task.Jun 22 2020, 9:15 AM
Restricted Application added a project: User-Addshore. · View Herald TranscriptJun 22 2020, 9:15 AM
Krinkle moved this task from Untriaged to Jun 2020 on the Wikimedia-production-error board.Jun 24 2020, 6:55 PM
Addshore merged a task: T252061: [data-update-failed]: Avoiding infinite loop due to possibily undetectable existing records (Wikibase DataUpdateAdapter).Jun 30 2020, 8:58 AM
Addshore added a subscriber: brennen.
gerritbot added a comment.Jun 30 2020, 9:00 AM

Change 606692 abandoned by Addshore:
AdHocLogging for ReplicaMasterAwareRecordIdsAcquirer

Reason:
No longer needed to backport

https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Wikibase/ /606692

Addshore moved this task from Doing to Peer Review on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.Jun 30 2020, 7:54 PM
gerritbot added a comment.Jul 1 2020, 4:47 PM

Change 606691 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] More logging for ReplicaMasterAwareRecordIdsAcquirer error case

https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Wikibase/ /606691

Michael moved this task from Peer Review to Test (Verification) on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.Jul 1 2020, 4:58 PM
ReleaseTaggerBot added a project: MW-1.35-notes (1.35.0-wmf.40; 2020-07-07).Jul 1 2020, 5:00 PM
Addshore moved this task from Unsorted 💣 to Closing ✔️ on the User-Addshore board.Jul 6 2020, 10:17 PM
Addshore closed this task as Resolved.Jul 16 2020, 7:15 AM
Addshore moved this task from Test (Verification) to Done on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.

Marking as resolved as the issue and spike went away, and we added more logging

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits