Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Multiple bug fixes, improvements, and addtions. #3

Open
wants to merge 27 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
27 commits
Select commit Hold shift + click to select a range
afb2463
bugfix for id_list parameter
ota42y May 18, 2020
3c08a31
Updated .gitignore
marvin-hansen Feb 21, 2022
b6dfad2
Updated Readme.
marvin-hansen Feb 21, 2022
f56ca1a
Update README.md
marvin-hansen Feb 21, 2022
c942631
Moved atom structs to internal structs
marvin-hansen Feb 21, 2022
28ef865
Merge remote-tracking branch 'origin/master'
marvin-hansen Feb 21, 2022
e48de75
Update README.md
marvin-hansen Feb 21, 2022
c8ae66a
Moved atom structs to internal structs
marvin-hansen Feb 21, 2022
480b92d
Update README.md
marvin-hansen Feb 21, 2022
12c39f4
Updated feed to return all authors.
marvin-hansen Feb 21, 2022
17370b6
Update README.md
marvin-hansen Feb 21, 2022
9fbfec5
Changed Person field to struct of pointers
marvin-hansen Feb 21, 2022
981cc86
Merge remote-tracking branch 'origin/master'
marvin-hansen Feb 21, 2022
7e353fb
Update README.md
marvin-hansen Feb 21, 2022
7cc002a
Added missing category field to Entry struct.
marvin-hansen Feb 21, 2022
d29c36e
Merge remote-tracking branch 'origin/master'
marvin-hansen Feb 21, 2022
8f43d3b
chanted category type to string
marvin-hansen Feb 21, 2022
d6a7c4b
Added primary category of an entry.
marvin-hansen Feb 21, 2022
b9a79dc
Working on parsing xml attributes.
marvin-hansen Feb 21, 2022
98a9710
Working on parsing xml attributes.
marvin-hansen Feb 21, 2022
da4bfab
Update README.md
marvin-hansen Feb 21, 2022
fbbff8b
Improved XML annotation.
marvin-hansen Feb 21, 2022
fd38aea
Merge remote-tracking branch 'origin/master'
marvin-hansen Feb 21, 2022
59fb5ed
Update README.md
marvin-hansen Feb 21, 2022
938d135
Added missing categories:
marvin-hansen Feb 22, 2022
c141a01
Update README.md
marvin-hansen Feb 22, 2022
926e6ec
Update README.md
marvin-hansen Feb 22, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
46 changes: 27 additions & 19 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,24 +1,32 @@
# Compiled Object files, Static and Dynamic libs (Shared Objects)
*.o
*.a
*.so
.idea
.idea/*

# Folders
_obj
_test
.classpath
.settings/
.groovy/
.ijwb/

# Architecture specific extensions/prefixes
*.[568vq]
[568vq].out
notes/*
notes/*.*

*.cgo1.go
*.cgo2.c
_cgo_defun.c
_cgo_gotypes.go
_cgo_export.*
# Bazel stuff
bazel-out
bazel-out/*.*
dist
dist/*.*

_testmain.go
# Emacs save files
*~
\#*\#
.\#*

*.exe
*.test
*.prof
# Vim-related files
[._]*.s[a-w][a-z]
[._]s[a-w][a-z]
*.un~
Session.vim
.netrwhist

# Crash & log files
crash.log
output.log
18 changes: 16 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,19 +1,33 @@
# arxiv

Patched version of the stalled go-lang client.
* Fixed: id_list parameter.
* Fixed: Returns all authors now.
* Fixed: Returns primary cateegory
* Fixed: Retruns all secondary categories
* Fixed: Decodes category keys (i.e. math.SP) to a human readable format (i.e. MathsSpectralTheory) using categeory.Term.String() method
* Fixed: Added missing categories in major fields: Economics, Quantitative Finance, Statistics, & Electrical Engineering and Systems Science.

Go API client for arxiv.org. It supports simple as well as advanced searches with filters.


## Install
```go
go get github.com/marvin-hansen/[email protected]
```

## Usage
Sample usage can be found in file [example_test.go](./example_test.go)
Or see below:
* Preamble:
* Import:
```go
package main

import (
"fmt"
"log"

"github.com/orijtech/arxiv/v1"
"github.com/marvin-hansen/arxiv/v1"
)
```

Expand Down
2 changes: 0 additions & 2 deletions example_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -18,8 +18,6 @@ import (
"context"
"fmt"
"log"

"github.com/orijtech/arxiv/v1"
)

func Example_client_Search_simple() {
Expand Down
26 changes: 12 additions & 14 deletions v1/arxiv.go
Original file line number Diff line number Diff line change
Expand Up @@ -19,17 +19,14 @@ import (
"encoding/xml"
"errors"
"fmt"
"github.com/orijtech/otils"
"go.opencensus.io/plugin/ochttp"
"go.opencensus.io/trace"
"io/ioutil"
"net/http"
"strings"
"sync"
"time"

"golang.org/x/tools/blog/atom"

"go.opencensus.io/plugin/ochttp"
"go.opencensus.io/trace"

"github.com/orijtech/otils"
)

type Client struct {
Expand Down Expand Up @@ -107,10 +104,9 @@ type Query struct {
// Fresh struct here to avoid sending
// unnecessary query string values from Query
type reqPage struct {
Terms string `json:"search_query"`
Start int64 `json:"start"`
MaxResults int64 `json:"max_results"`
ArticleIDs []string `json:"id_list"`
Terms string `json:"search_query"`
Start int64 `json:"start"`
MaxResults int64 `json:"max_results"`

SortBy SortBy `json:"sortBy"`
SortOrder SortOrder `json:"sortOrder"`
Expand All @@ -122,7 +118,6 @@ func (q *Query) reqPage() *reqPage {
Start: q.PageNumber,
SortBy: q.SortBy,
MaxResults: q.MaxResultsPerPage,
ArticleIDs: q.ArticleIDs[:],
SortOrder: q.SortOrder,
}
}
Expand All @@ -143,7 +138,7 @@ func (q *Query) Validate() error {
}

type ResultsPage struct {
Feed *atom.Feed `json:"feed"`
Feed *Feed `json:"feed"`

PageNumber int64 `json:"page_number"`

Expand Down Expand Up @@ -220,6 +215,9 @@ func (c *Client) Search(ctx context.Context, q *Query) (chan *ResultsPage, cance
if len(filters) > 0 {
qv.Set("search_query", filtersStr)
}
if len(q.ArticleIDs) > 0 {
qv.Set("id_list", strings.Join(q.ArticleIDs, ","))
}

cctx, span := trace.StartSpan(ctx, "paging")
span.Annotate([]trace.Attribute{
Expand All @@ -243,7 +241,7 @@ func (c *Client) Search(ctx context.Context, q *Query) (chan *ResultsPage, cance
}

_, umSpan := trace.StartSpan(ctx, "xml_unmarshal-to-atom.Feed")
feed := new(atom.Feed)
feed := new(Feed)
err = xml.Unmarshal(slurp, feed)
umSpan.End()

Expand Down
6 changes: 1 addition & 5 deletions v1/arxiv_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -23,10 +23,6 @@ import (
"os"
"strconv"
"testing"

"golang.org/x/tools/blog/atom"

"github.com/orijtech/arxiv/v1"
)

func TestSearch(t *testing.T) {
Expand Down Expand Up @@ -164,7 +160,7 @@ func resultsPageFromFile(searchTerm string, pageNumber int64) *arxiv.ResultsPage
if err != nil {
return nil
}
feed := new(atom.Feed)
feed := new(Feed)
if err := xml.Unmarshal(blob, feed); err != nil {
return nil
}
Expand Down
384 changes: 384 additions & 0 deletions v1/category.go

Large diffs are not rendered by default.

66 changes: 66 additions & 0 deletions v1/feed.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
package arxiv

import (
"encoding/xml"
"time"
)

type Feed struct {
XMLName xml.Name `xml:"http://www.w3.org/2005/Atom feed"`
Title string `xml:"title"`
ID string `xml:"id"`
Link []Link `xml:"link"`
Updated TimeStr `xml:"updated"`
Author []*Person `xml:"author"`
Entry []*Entry `xml:"entry"`
}

type Entry struct {
XMLName xml.Name
Doi string `xml:"doi"`
Title string `xml:"title"`
ID string `xml:"id"`
Link []Link `xml:"link"`
Published TimeStr `xml:"published"`
Updated TimeStr `xml:"updated"`
Comment string `xml:"comment"`
Author []*Person `xml:"author"`
Summary *Text `xml:"summary"`
Content *Text `xml:"content"`
PrimaryCategory *Class `xml:"primary_category,omitempty"`
Category []*Class `xml:"category,omitempty"`
}

type Class struct {
XMLName xml.Name
Term Category `xml:"term,attr"`
}

type Link struct {
XMLName xml.Name
Rel string `xml:"rel,attr,omitempty"`
Href string `xml:"href,attr"`
Type string `xml:"type,attr,omitempty"`
HrefLang string `xml:"hreflang,attr,omitempty"`
Title string `xml:"title,attr,omitempty"`
Length uint `xml:"length,attr,omitempty"`
}

type Person struct {
XMLName xml.Name
Name string `xml:"name"`
URI string `xml:"uri,omitempty"`
Email string `xml:"email,omitempty"`
}

type Text struct {
XMLName xml.Name
Type string `xml:"type,attr"`
Body string `xml:",chardata"`
}

type TimeStr string

func Time(t time.Time) TimeStr {
return TimeStr(t.Format("2006-01-02T15:04:05-07:00"))
}
134 changes: 0 additions & 134 deletions v1/filters.go
Original file line number Diff line number Diff line change
Expand Up @@ -160,137 +160,3 @@ func jsonTag(v reflect.StructField) (tag string, omitempty, ignore bool) {
_, ignore = instrIndex["-"]
return tag, omitempty, ignore
}

// Category

type Category string

const (
StatisticsApplications Category = "stat.AP"
StatisticsComputation Category = "stat.CO"
StatisticsMachineLearning Category = "stat.ML"
StatisticsMethodology Category = "stat.ME"
StatisticsTheory Category = "stat.TH"
QuantitativeBiologyBiomolecules Category = "q-bio.BM"
QuantitativeBiologyCellBehavior Category = "q-bio.CB"
QuantitativeBiologyGenomics Category = "q-bio.GN"
QuantitativeBiologyMolecularNetworks Category = "q-bio.MN"
QuantitativeBiologyNeuronsAndCognition Category = "q-bio.NC"
QuantitativeBiologyOther Category = "q-bio.OT"
QuantitativeBiologyPopulationsAndEvolution Category = "q-bio.PE"
QuantitativeBiologyQuantitativeMethods Category = "q-bio.QM"
QuantitativeBiologySubcellularProcesses Category = "q-bio.SC"
QuantitativeBiologyTissuesAndOrgans Category = "q-bio.TO"
CSArchitecture Category = "cs.AR"
CSArtificialIntelligence Category = "cs.AI"
CSComputationAndLanguage Category = "cs.CL"
ComputationalComplexity Category = "cs.CC"
ComputationalEngineeringFinanceAndScience Category = "cs.CE"
CSComputationalGeometry Category = "cs.CG"
CSGameTheory Category = "cs.GT"
ComputerVisionAndPatternRecognition Category = "cs.CV"

ComputersAndSociety Category = "cs.CY"
CryptographyAndSecurity Category = "cs.CR"
Databases Category = "cs.DB"
DigitalLibraries Category = "cs.DL"
DiscreteMathematics Category = "cs.DM"
DistributedParallelAndClusterComputing Category = "cs.DC"
CSGeneralLiterature Category = "cs.GL"
CSGraphics Category = "cs.GR"
HumanComputerInteraction Category = "cs.HC"
CSInformationRetrieval Category = "cs.IR"
CSInformationTheory Category = "cs.IT"
CSLearning Category = "cs.LG"
CSLogic Category = "cs.LO"
CSMathematicalSoftware Category = "cs.MS"
MultiagentSystems Category = "cs.MA"
CSMultimedia Category = "cs.MM"
NetworkAndInternetArchitecture Category = "cs.NI"
NeuralAndEvolutionaryComputing Category = "cs.NE"
CSNumericalAnalysis Category = "cs.NA"
OperatingSystems Category = "cs.OS"
CSOther Category = "cs.OH"
CSPerformance Category = "cs.PF"
ProgrammingLanguages Category = "cs.PL"
CSRobotics Category = "cs.RO"
SoftwareEngineering Category = "cs.SE"
CSSound Category = "cs.SD"
SymbolicComputation Category = "cs.SC"
NonLinearSciencesAdaptationAndSelfOrganizingSystemsCategory = "nlin.AO"
NonLinearSciencesCellularAutomataAndLatticeGases Category = "nlin.CG"
NonLinearSciencesChaoticDynamics Category = "nlin.CD"
ExactlySolvableAndIntegrableSytems Category = "nlin.SI"
PatternFormationAndSolutions Category = "nlin.PS"
AlgebraicGeometry Category = "math.AG"
AlgebraicTopology Category = "math.AT"
AnalysisOfPDEs Category = "math.AP"
CategoryTheory Category = "math.CT"
ClassicalAnalysisAndODEs Category = "math.CA"
Combinatorics Category = "math.CO"
CommutativeAlgebra Category = "math.AC"
ComplexVariables Category = "math.CV"
DifferentialGeometry Category = "math.DG"
DynamicalSystems Category = "math.DS"
FunctionalAnalysis Category = "math.FA"
GeneralMathematics Category = "math.GM"
GeneralTopology Category = "math.GN"
GeometricTopology Category = "math.GT"
GroupTheory Category = "math.GR"
MathsHistoryAndOverview Category = "math.HO"
MathsInformationTheory Category = "math.IT"
KTheoryAndHomology Category = "math.KT"
MathsLogic Category = "math.LO"
MathsMathematicalPhysics Category = "math.MP"
MetricGeometry Category = "math.MG"
NumberTheory Category = "math.NT"
MathsNumericalAnalysis Category = "math.NA"
OperatorAlgebras Category = "math.OA"
MathsOptimizationAndControl Category = "math.OC"
Probability Category = "math.PR"
QuantumAlgebra Category = "math.QA"
RepresentationTheory Category = "math.RT"
RingsAndAlgebra Category = "math.RA"
MathsSpectralTheory Category = "math.SP"
MathsStatics Category = "math.ST"
SymplecticGeometry Category = "math.SG"
Astrophysics Category = "astro-ph"
PhysicsDisorderedSystemsAndNeuralNetworks Category = "cond-mat.dis-nn"
PhysicsMesoscopicSystemsAndQuantumHallEffect Category = "cond-mat.mes-hall"
PhysicsMaterialsScience Category = "cond-mat.mtrl-sci"
PhysicsOther Category = "cond-mat.other"
PhysicsSoftCondensedMatter Category = "cond-mat.soft"
PhysicsStatisticalMechanics Category = "cond-mat.stat-mech"
PhysicsStronglyCorrelatedElectrons Category = "cond-mat.str-el"
PhysicsSuperconductivity Category = "cond-mat.sup-con"
GeneralRelativityAndQuantumCosmology Category = "gr-qc"
HighEneryPhysicsExperiment Category = "hep-ex"
HighEneryPhysicsLattice Category = "hep-lat"
HighEneryPhysicsPhenomenology Category = "hep-ph"
HighEneryPhysicsTheory Category = "hep-th"
MathematicalPhysics Category = "math-ph"
NuclearExperiment Category = "nucl-ex"
NuclearTheory Category = "nucl-th"
AcceleratorPhysics Category = "physics.acc-ph"
AtmoshpericAndOceanicPhysics Category = "physics.ao-ph"
AtomicPhysics Category = "physics.atom-ph"
AtomicAndMolecularClusters Category = "physics.atm-clus"
BiologicalPhysics Category = "physics.bio-ph"
ChemicalPhysics Category = "physics.chem-ph"
ClassicalPhysics Category = "physics.class-ph"
ComputationalPhysics Category = "physics.comp-ph"
DataAnalysisStatisticsAndProbability Category = "physics.data-an"
FluidDynamics Category = "physics.flu-dyn"
GeneralPhysics Category = "physics.gen-ph"
Geophysics Category = "physics.geo-ph"
HistoryOfPhysics Category = "physics.hist-ph"
InstrumentationAndDetectors Category = "physics.ins-det"
MedicalPhysics Category = "physics.med-ph"
Optics Category = "physics.optics"
PhysicsEducation Category = "physics.ed-ph"
PhysicsAndSociety Category = "physics.soc-ph"
PlasmaPhysics Category = "physiscs.plasm-ph"
PopularPhysics Category = "physics.pop-ph"
SpacePhysics Category = "physics.space-ph"
QuantumPhysics Category = "quant-ph"
)
Loading