Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

run_ner.py raised error #10006

Closed
gongel opened this issue Feb 4, 2021 · 11 comments
Closed

run_ner.py raised error #10006

gongel opened this issue Feb 4, 2021 · 11 comments

Comments

@gongel
Copy link

gongel commented Feb 4, 2021

Environment info

  • transformers version: 4.3.0.dev0
  • Platform: MacOS
  • Python version: 3.6
  • PyTorch version (GPU?): 1.7.1
  • Tensorflow version (GPU?): 2.4.1
  • Using GPU in script?: No
  • Using distributed or parallel set-up in script?: No

Who can help

Examples:

To reproduce

Steps to reproduce the behavior:

  1. bash run.sh to run_ner.py

Error

ReferenceError: {'help': 'The name of the task (ner, pos...).'} does not reference a class __dict__
@patil-suraj
Copy link
Contributor

I just ran run.sh, but did not see this error. Could you maybe post the full stack trace?

@gongel
Copy link
Author

gongel commented Feb 4, 2021

Hi @patil-suraj, thanks for replying. The full stack trace is posted below.

Traceback (most recent call last):
  File "run_origin.py", line 437, in <module>
    main()
  File "run_origin.py", line 310, in main
    load_from_cache_file=not data_args.overwrite_cache,
  File "/Users/bytedance/opt/anaconda3/lib/python3.6/site-packages/datasets/dataset_dict.py", line 303, in map
    for k, dataset in self.items()
  File "/Users/bytedance/opt/anaconda3/lib/python3.6/site-packages/datasets/dataset_dict.py", line 303, in <dictcomp>
    for k, dataset in self.items()
  File "/Users/bytedance/opt/anaconda3/lib/python3.6/site-packages/datasets/arrow_dataset.py", line 1259, in map
    update_data=update_data,
  File "/Users/bytedance/opt/anaconda3/lib/python3.6/site-packages/datasets/arrow_dataset.py", line 157, in wrapper
    out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
  File "/Users/bytedance/opt/anaconda3/lib/python3.6/site-packages/datasets/fingerprint.py", line 158, in wrapper
    self._fingerprint, transform, kwargs_for_fingerprint
  File "/Users/bytedance/opt/anaconda3/lib/python3.6/site-packages/datasets/fingerprint.py", line 105, in update_fingerprint
    hasher.update(transform_args[key])
  File "/Users/bytedance/opt/anaconda3/lib/python3.6/site-packages/datasets/fingerprint.py", line 57, in update
    self.m.update(self.hash(value).encode("utf-8"))
  File "/Users/bytedance/opt/anaconda3/lib/python3.6/site-packages/datasets/fingerprint.py", line 53, in hash
    return cls.hash_default(value)
  File "/Users/bytedance/opt/anaconda3/lib/python3.6/site-packages/datasets/fingerprint.py", line 46, in hash_default
    return cls.hash_bytes(dumps(value))
  File "/Users/bytedance/opt/anaconda3/lib/python3.6/site-packages/datasets/utils/py_utils.py", line 389, in dumps
    dump(obj, file)
  File "/Users/bytedance/opt/anaconda3/lib/python3.6/site-packages/datasets/utils/py_utils.py", line 361, in dump
    Pickler(file, recurse=True).dump(obj)
  File "/Users/bytedance/opt/anaconda3/lib/python3.6/pickle.py", line 409, in dump
    self.save(obj)
  File "/Users/bytedance/opt/anaconda3/lib/python3.6/pickle.py", line 476, in save
    f(self, obj) # Call unbound method with explicit self
  File "/Users/bytedance/opt/anaconda3/lib/python3.6/site-packages/datasets/utils/py_utils.py", line 556, in save_function
    obj=obj,
  File "/Users/bytedance/opt/anaconda3/lib/python3.6/pickle.py", line 610, in save_reduce
    save(args)
  File "/Users/bytedance/opt/anaconda3/lib/python3.6/pickle.py", line 476, in save
    f(self, obj) # Call unbound method with explicit self
  File "/Users/bytedance/opt/anaconda3/lib/python3.6/pickle.py", line 751, in save_tuple
    save(element)
  File "/Users/bytedance/opt/anaconda3/lib/python3.6/pickle.py", line 476, in save
    f(self, obj) # Call unbound method with explicit self
  File "/Users/bytedance/opt/anaconda3/lib/python3.6/pickle.py", line 751, in save_tuple
    save(element)
  File "/Users/bytedance/opt/anaconda3/lib/python3.6/pickle.py", line 476, in save
    f(self, obj) # Call unbound method with explicit self
  File "/Users/bytedance/opt/anaconda3/lib/python3.6/site-packages/dill/_dill.py", line 1129, in save_cell
    pickler.save_reduce(_create_cell, (f,), obj=obj)
  File "/Users/bytedance/opt/anaconda3/lib/python3.6/pickle.py", line 610, in save_reduce
    save(args)
  File "/Users/bytedance/opt/anaconda3/lib/python3.6/pickle.py", line 476, in save
    f(self, obj) # Call unbound method with explicit self
  File "/Users/bytedance/opt/anaconda3/lib/python3.6/pickle.py", line 736, in save_tuple
    save(element)
  File "/Users/bytedance/opt/anaconda3/lib/python3.6/pickle.py", line 521, in save
    self.save_reduce(obj=obj, *rv)
  File "/Users/bytedance/opt/anaconda3/lib/python3.6/pickle.py", line 605, in save_reduce
    save(cls)
  File "/Users/bytedance/opt/anaconda3/lib/python3.6/pickle.py", line 476, in save
    f(self, obj) # Call unbound method with explicit self
  File "/Users/bytedance/opt/anaconda3/lib/python3.6/site-packages/dill/_dill.py", line 1315, in save_type
    obj.__bases__, _dict), obj=obj)
  File "/Users/bytedance/opt/anaconda3/lib/python3.6/pickle.py", line 610, in save_reduce
    save(args)
  File "/Users/bytedance/opt/anaconda3/lib/python3.6/pickle.py", line 476, in save
    f(self, obj) # Call unbound method with explicit self
  File "/Users/bytedance/opt/anaconda3/lib/python3.6/pickle.py", line 751, in save_tuple
    save(element)
  File "/Users/bytedance/opt/anaconda3/lib/python3.6/pickle.py", line 476, in save
    f(self, obj) # Call unbound method with explicit self
  File "/Users/bytedance/opt/anaconda3/lib/python3.6/site-packages/dill/_dill.py", line 902, in save_module_dict
    StockPickler.save_dict(pickler, obj)
  File "/Users/bytedance/opt/anaconda3/lib/python3.6/pickle.py", line 821, in save_dict
    self._batch_setitems(obj.items())
  File "/Users/bytedance/opt/anaconda3/lib/python3.6/pickle.py", line 847, in _batch_setitems
    save(v)
  File "/Users/bytedance/opt/anaconda3/lib/python3.6/pickle.py", line 476, in save
    f(self, obj) # Call unbound method with explicit self
  File "/Users/bytedance/opt/anaconda3/lib/python3.6/site-packages/dill/_dill.py", line 902, in save_module_dict
    StockPickler.save_dict(pickler, obj)
  File "/Users/bytedance/opt/anaconda3/lib/python3.6/pickle.py", line 821, in save_dict
    self._batch_setitems(obj.items())
  File "/Users/bytedance/opt/anaconda3/lib/python3.6/pickle.py", line 847, in _batch_setitems
    save(v)
  File "/Users/bytedance/opt/anaconda3/lib/python3.6/pickle.py", line 521, in save
    self.save_reduce(obj=obj, *rv)
  File "/Users/bytedance/opt/anaconda3/lib/python3.6/pickle.py", line 634, in save_reduce
    save(state)
  File "/Users/bytedance/opt/anaconda3/lib/python3.6/pickle.py", line 476, in save
    f(self, obj) # Call unbound method with explicit self
  File "/Users/bytedance/opt/anaconda3/lib/python3.6/pickle.py", line 736, in save_tuple
    save(element)
  File "/Users/bytedance/opt/anaconda3/lib/python3.6/pickle.py", line 476, in save
    f(self, obj) # Call unbound method with explicit self
  File "/Users/bytedance/opt/anaconda3/lib/python3.6/site-packages/dill/_dill.py", line 902, in save_module_dict
    StockPickler.save_dict(pickler, obj)
  File "/Users/bytedance/opt/anaconda3/lib/python3.6/pickle.py", line 821, in save_dict
    self._batch_setitems(obj.items())
  File "/Users/bytedance/opt/anaconda3/lib/python3.6/pickle.py", line 847, in _batch_setitems
    save(v)
  File "/Users/bytedance/opt/anaconda3/lib/python3.6/pickle.py", line 476, in save
    f(self, obj) # Call unbound method with explicit self
  File "/Users/bytedance/opt/anaconda3/lib/python3.6/site-packages/dill/_dill.py", line 1148, in save_dictproxy
    raise ReferenceError("%s does not reference a class __dict__" % obj)
ReferenceError: {'help': 'The name of the task (ner, pos...).'} does not reference a class __dict__

@patil-suraj
Copy link
Contributor

looks like you are running your own script run_origin.py, so the issue is not with run_ner.py

@stefan-it
Copy link
Collaborator

stefan-it commented Feb 4, 2021 •
edited
Loading

Hey @gongel ,

could you confirm that you are really using latest 4.3 version 🤔

For me the example is working withrun_ner.sh.

@gongel
Copy link
Author

gongel commented Feb 4, 2021

@patil-suraj, I just renamed run_ner.py to run_origin.py.

@gongel
Copy link
Author

gongel commented Feb 4, 2021

Hi @stefan-it , Yes

(base) C02D925LMD6R:transformers gong$ pip show transformers
Name: transformers
Version: 4.3.0.dev0
Summary: State-of-the-art Natural Language Processing for TensorFlow 2.0 and PyTorch
Home-page: https://github.com/huggingface/transformers
Author: Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Sam Shleifer, Patrick von Platen, Sylvain Gugger, Google AI Language Team Authors, Open AI team Authors, Facebook AI Authors, Carnegie Mellon University Authors
Author-email: [email protected]
License: Apache
Location: /Users/bytedance/transformers/src
Requires: filelock, numpy, packaging, regex, requests, sacremoses, tokenizers, tqdm, dataclasses, importlib-metadata
Required-by: sentence-transformers

@patil-suraj
Copy link
Contributor

Could you run the run_ner.py script using master? as Stefan said your version seems old.

@gongel
Copy link
Author

gongel commented Feb 5, 2021

I tried 4.3.0.dev0, 4.4.0.dev0 and 4.2.2 .
They all didn't work. 😭

@sgugger
Copy link
Collaborator

sgugger commented Feb 5, 2021

You might have an issue if your version of datasets is old. In any case, the whole serialization error is linked to the datasets library, so pinging @lhoestq in case he has a better idea :-)

@lhoestq
Copy link
Member

lhoestq commented Feb 5, 2021

Hi !
Can you try updating dill ?
It looks like one of their issues from 2019 that has been fixed now.

@gongel
Copy link
Author

gongel commented Feb 5, 2021

Thank you, @lhoestq @sgugger @patil-suraj @stefan-it
It works by updating dill from 0.2.9 to 0.3.3.

@gongel gongel closed this as completed Feb 5, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@stefan-it @gongel @patil-suraj @sgugger @lhoestq