Do Surgeons Agree on Severity and Origin of Complications in Bone-lengthening Nails? An Inter- and Intra-rater Reliability Study

INTRODUCTION

Fully implantable intramedullary lengthening nails were developed to overcome or reduce the complications and patient discomfort in bone-lengthening using conventional external fixators.¹ However, a systematic review has demonstrated a high rate of complications with bone-lengthening nails.² In a select group of patients operated by highly experienced limb lengthening surgeons, 76% of patients sustained a complication,³ and complications become even more frequent in bone transport nails.⁴ The severity of complications differs substantially from minor complications that can be easily handled without additional surgery to very severe complications imposing new and permanent sequelae on the patient.^2,4 We have identified five different ways to report the severity of complications with bone-lengthening nails: Four classification systems and one descriptive approach.^2,5–8 To the best of our knowledge, none of the reporting systems have so far been tested for reliability, and no gold standard exists. The need for a shared language on complications to assess and report the results accurately across patients was recently emphasised in the article by Sabharwal.⁹ We believe that the severity grading system by Black et al. is easy to use and, at the same time, classifies complications according to clinical relevant parameters based on the extent of therapeutic needs and their effect on the outcome.^8,10 It defines a “complication” as an unpredicted undesirable deviation from the treatment plan.⁸ Without appropriate intervention, it will lead to failure to achieve treatment goals or result in a new pathology.¹⁰ Complications are categorised into four categories based on severity. The system was first reported for patients with external fixation, making it possible to compare complications between bone lengthening with external fixation and fully implantable nails.^8,10 As complications arise from many different origins with bone-lengthening nails,² it is also essential to classify the origin. None of the classification systems assess the origin of the complication,^2,5–8 Therefore, we have developed a classification system where the origin is categorised into eight main groups and 33 subgroups.² To allow for the comparison of complications, the reliability of the reporting system must be known.¹¹ Information about complications can be extracted both from patient charts or from cases reported in the literature which can be compiled cross studies to gain knowledge in some of the highly specialised cases. The study aims to assess the inter- and intra-rater agreement of a classification system on complication severity grading and origin with bone-lengthening nails. The reliability was evaluated in two settings/parts: (a) Complications detected from patient charts in an observational cohort study and (b) complications retrieved from already published articles allocated through systematic literature searches.

MATERIALS AND METHODS

The study was divided into two settings/parts. The first part is composed of cohort cases. Following institutional approval at Aalborg University Hospital (AAUH), Denmark, a cohort of 279 segments (223 patients) treated with bone-lengthening nails at the lower extremity was identified in the period between 2005 and 2021. Two hundred sixty complications were acknowledged through the patient charts assessment of the cohort. The main author selected 49 complications from the patient charts to represent severities and origins (one case was excluded due to a double-entry error).

The second part was composed of published literature cases. A systematic literature search found 952 articles, and the assessment selected 41 papers that contained 782 patients with 332 complications (search string and article selection see Frost et al.)² Forty-nine of those complications were selected by the main author representing spectrum of severities and origins among the complications. Only cases with magnetic PRECICE® (NUVASIVE, San Diego, California, USA) and electric FITBONE® (WITTENSTEIN intens GmbH, Igersheim, Germany; distributed by Orthofix Medical, Inc., Lewisville, Texas, USA) bone-lengthening nails were included, and for both the literature and cohort parts, both nail’ types were represented.

A Research Electronic Data Capture (REDCap) database managed the cases and ratings. Complication information was captured in the REDCap database: For literature cases, the article description was copied, and for cohort cases, the relevant charts information for scoring complications was copied from the patients’ charts in an anonymous form. The raters were presented with the exact text from papers or charts in each of the two study parts in the database.

A pre-study workshop with the four raters attending was established to reach a consensus on the interpretation of the classification system. The study background and method were introduced in combination with the education of the classification system. Seven test-literature cases and five test-cohort cases were reviewed at the pre-study workshop. These test-study cases were not included in the study. The data collection tool, a redcap database, was introduced. A written guideline was agreed upon. In general, if the treatment or the outcome of a complication was not thoroughly described, we downgraded the severity of the complication.

The REDCap data collection instrument was designed with push buttons to make the assessment complete. The database had two different parts to classify the complication into severity and origin from a text box describing the case. All raters were instructed to perform the ratings independently, and the rater was blinded for other raters’ assessments and their own previous ratings. Ratings of all cases were performed twice by all raters at least six weeks apart. All cases were randomly switched around between the first and second ratings to avoid recall of the ratings due to the order of the cases. The raters were informed that their ratings would be compared and published before the assessment of the cases.

The raters consisted of two senior orthopaedic surgeons specialised in reconstructive surgery: One consultant with 8 years of experience in bone lengthening (rater A: MM) and one consultant and clinical professor with more than 10 years of experience in bone lengthening (rater D: SK). One orthopaedic surgeon with 2 years of paediatric subspeciality and knowledge of handling bone-lengthening nail patients (rater B: MF). One orthopaedic surgeon with detailed research knowledge of complications with bone-lengthening nails but without clinical bone-lengthening experience (rater C: MWF). The raters’ experience with the severity and origin classification Raters D and C had extensive experience with the severity and origin classification from a review study.² Rater A had minor experience with severity classification from a small cohort study.⁴ Before the pre-study workshop, rater B had no experience with the severity and origin classification.

Complication severity was classified into four categories after Black et al.: I, II, IIIA and IIIB.⁸ The severity classification is presented in Table 1. The following complications were defined as type IIIB: A deep vein thrombosis, osteomyelitis and joint subluxation/luxation. The type of complication was categorised into origins representing eight main groups (soft tissue, joint, vascular, bone, neurological, infection, device related and others) and 33 subgroups according to Frost et al.² The following definitions were applied when categorising the origin: If no device failure or bone failure occurred during treatment, a residual deformity or an intra-articular nail protrusion causing joint pathologies were classified into Others/Surgical. A patient request to stop the lengthening procedure without other reason was categorised into Others/Patient. Neuralgia was categorised into Neurology/Others. Compartment syndrome was categorised into Soft tissue/Others. Pain from a locking screw or a subcutaneous receiver in an electric lengthening nail was categorised into Device-related/Others.

RESULTS

In the cohort cases, the orthopaedic surgeon’s inter-rater kappa value agreement was 0.68 (CI: 0.56–0.79) for severity and 0.63 (CI: 0.53–0.73) for the main origin with subtype classification, indicating a good agreement with a CI upper limit of excellent agreement. Table 2 presents the inter-rater kappa and observed agreement estimates from the cohort study part. Raters C and D had the highest agreement grade, whereas raters A and B had the lowest.

An inter-rater kappa value of 0.64 (CI: 0.53–0.75) for severity was observed in the literature cases, indicating good agreement. The observed agreement and inter-rater kappa values from literature cases are presented in Table 3. For the main origin, a kappa value of 0.80 [0.67:0.94] for the inter-rater agreement was observed corresponding to an excellent agreement with a CI ranging from good to excellent agreement. The kappa value for the main origin with subtype was 0.74 [CI: 0.64–0.83], corresponding to good or excellent agreement. Comparing the severity grading of complications between individual raters, we found that raters C and D had the highest pairwise inter-rater agreement (Tables 2 and 3). A similar pattern was seen for main and subtype origin; however, the kappa values were higher for origin classification than for severity grading.

For the cohort cases, the overall intra-rater agreement of the four orthopaedic surgeons had kappa values ranging from 0.70–0.77 on severity and origin with subtype classification, corresponding to the good or excellent agreement (Table 4). The kappa agreement estimate for rater C was excellent for severity and origin classification, and for the other raters, the agreements ranged from good to excellent depending on severity and origin classification in the cohort study part.

The intra-rater kappa value of the complication severity classification, in the literature cases, was 0.78 (CI: 0.70–0.85), corresponding to an excellent agreement, with a interval indicating good to excellent agreement. Table 5 presents the intra-rater kappa values and observed agreement estimates from the literature study part. We observed an excellent intra-rater agreement of the main origin complication type and excellent or good for the main plus subtype origin in the literature cases. Rater C showed excellent intra-rater agreement, and the other raters showed from poor to excellent intra-rater agreement depending on severity and origin classification.

DISCUSSION

To our knowledge, this study is the first to examine the reliability of a classification system for reporting on the severity and the origin of complications with bone-lengthening nails. The study has demonstrated that the severity and the origin complications classification had an overall good reliability agreement.

A shared complication classification system is important to assess and report the results of bone-lengthening nails accurately across patients and studies.^9,15 We have previously detected four different systems for reporting on the severity of complications with bone-lengthening nails.² In the current study, we examined the severity classification proposed by Black et al.⁸ We believe that this severity classification is simple to use and that it applies a clinically relevant definition of a complication being an unpredicted undesirable deviation from the treatment plan, which without appropriate resolution will lead to a failure to achieve treatment goals or to the development of a new pathology.¹⁰ Furthermore, the grading is based only on the severity of the complications without relation to the time point in the treatment. In the literature on bone lengthening, Paley’s three-step classification system “Problem, Obstacle and Complication” are often used.⁵ Problems and obstacles occur during distraction or consolidation and are resolved by the end of the treatment by nonoperative and operative matter, respectively.⁵ A complication can occur from intraoperative to the end of treatment and is not resolved at the end of treatment.⁵ A minor complication is a delay in treatment or rehabilitation of an annoying problem leaving no significant residua, and a major complication interferes with achieving the treatment goals.⁵ Dahl et al. also used a three-step classification with minor, serious and severe complications.⁷ Minor was defined as a complication that did not affect the outcome and did not require intervention.⁷ A complication that was permanent and minor or transient and major was classified as serious.⁷ A severe complication resulted in permanent sequelae or unplanned surgery.⁷ A simpler classification with minor or major complications and true complications was used by Dinçyürek et al.⁶ The diversity in severity classification and their use of time points make a comparison between studies using different classification systems challenging. Another challenge in comparing complications is the lack of a common complication origin grouping. In a recent study by Frommer et al., adjustment of the distraction rate was the most frequent complication.³ It was, however, not noted if it was restricted joint motion, joint subluxation, pain, premature consolidation, or lack of bone regeneration that introduced the rate adjustment.³ In the classification of complications, we, therefore, combined the complications severity classification with a complications origin categorisation to obtain an understanding of complication origin impact on complication severity.

Observational cohort studies are predominantly the studies designed in the literature of intramedullary bone-lengthening nails, and therefore one of this study’s two parts was composed of cohort complication cases.² While many articles use a descriptive approach to reporting complications, the other part of this study comprised literature complications cases for the purpose of complication comparison between studies. For the two study parts of cohort and literature cases, a good overall inter-rater agreement on the main origin with subtype was observed with kappa values of 0.63 (CI: 0.53:0.73) and 0.74 (CI: 0.64:0.83), respectively. The higher kappa value of the literature cases could reflect that the assessment is simpler when a complication has been fitted into a published article compared to raw text in patient charts. The main and subtype assessment of cohort cases might be improved with a more detailed description of the subtypes.

Even though a pre-study workshop with rating training was held, and written instruction was provided, we observed an overall inter-rater agreement between orthopaedic surgeons on severity classification ranging from 0.64 and 0.68, corresponding to a good agreement. However, the inter-rater agreements between the two raters did show a high degree of variation with kappa values between 0.49 and 0.94. It seems that there is a learning curve for classifying complications since the highest inter-rater agreement was found between the two raters (C and D) with the highest classification experience. The variation of the intra-rater agreements both for cohort and literature cases further demonstrates that there is a learning curve when using this classification system. Rater C, who had substantial experience from a previous systematic review and a cohort study, demonstrated the highest intra-rater agreement.

The variance of the different raters’ clinical experience and expertise with the complication classification could reduce the study’s internal validity. Thus, a group consisting of only highly experienced raters might have achieved a higher agreement as opposed to our group consisting of raters with a high experience variance. With the treating clinicians (the two senior consultants) as raters in the study, there may be an assessment bias which would lead to a reduction in the inter-rater agreement. The choice of cases selected by the first author could induce a selection bias but have secured cases covering a wide range of severity and origin.

A further limitation of the study is the lack of the typical table presentation of the agreement prevalence, but the comprehensiveness of the possible outcomes of 4 raters, severity classification with four outcomes and origin classification with 33 categories makes this table an unworkable task and difficult to interpret. Therefore, we have presented the observed agreement with kappa estimates on all inter- and intra-rater estimates, and we have observed no signs of the two paradoxes of very low kappa values with a high observed agreement.¹⁶

The complications with bone-lengthening nails can have many different origins.² The categorisation into eight main categories and 33 subcategories reflects this complexity. However, the high number of categories affects the kappa value. As the number of categories increases, we expect a decrease in kappa value since the opportunity for misclassification or disagreement arises.¹¹

A strength of the applied classification system is that both the severity and the origin of a complication are categorised. By mapping both the origin and the severity of a classification, evidence-based interventions for reducing complications can be sought. We found that the classification system is reliable when used in a classic cohort case setting and applied to cases extracted from literature articles. The classification system of severity and origin can be adapted into bone lengthening with external fixation, which strengthens its future use. However, whether the same good agreement can be expected for classifying complications after external lengthening should be investigated in a separate study.

CONCLUSION AND CLINICAL SIGNIFICANCE

This agreement study is the first to test the reproducibility of a structured severity and origin classification of complications in intramedullary bone-lengthening nails in the settings of cohort and literature cases. For the cohort cases, the kappa estimates of 0.68 (CI: 0.56:0.79) and 0.63 (CI: 0.53:0.73) were observed for severity and origin, respectively. In the literature cases, the kappa estimate was 0.64 (CI: 0.53:0.75) and 0.74 (CI: 0.64:0.83) for severity and origin, respectively. Since a good inter-rater agreement was found in this study, the applied classification system may be suitable for reporting complications. A shared language for communicating complications for clinical and research purposes is important. The variation in an intra-rater agreement between reviewers indicates that future users should be focusing on improved reviewer guidelines and training before the use of the classification system. We encourage future studies to use a structured and validated complication classification to improve the shared understanding of complications with bone-lengthening nails.

ACKNOWLEDGEMENTS

The authors thank Kirsten Duch for her skillful and thorough biostatistics assistance.

ORCID

Markus Winther Frost https://orcid.org/0000-0002-5325-1415

Ole Rahbek https://orcid.org/0000-0002-5602-4533

Marie Fridberg https://orcid.org/0000-0002-8134-9444

Søren Kold https://orcid.org/0000-0002-3387-1473

REFERENCES

1. Simpson AHWR, Shalaby H, Keenan G. Femoral lengthening with the intramedullary skeletal kinetic distractor. J Bone Joint Surg Br 2009;91:955–961. DOI: 10.1302/0301-620X.91B7.21466.

2. Frost MW, Rahbek O, Traerup J, et al. Systematic review of complications with externally controlled motorized intramedullary bone lengthening nails (FITBONE and PRECICE) in 983 segments. Acta Orthop 2021;92:120–127. DOI: 10.1080/17453674.2020.1835321.

3. Frommer A, Roedl R, Gosheger G, et al. What are the potential benefits and risks of using magnetically driven antegrade intramedullary lengthening nails for femoral lengthening to treat leg length discrepancy? Clin Orthop Relat Res 2021;480(4):790–803. DOI: 10.1097/CORR.0000000000002036.

4. Mikužis M, Rahbek O, Christensen K, et al. Complications common in motorized intramedullary bone transport for non-infected segmental defects: A retrospective review of 15 patients. Acta Orthop 2012;92(4):485–492. DOI: 10.1080/17453674.2021.1910777.

5. Paley D. Problems, obstacles, and complications of limb lengthening by the Ilizarov technique. Clin Orthop Relat Res 1990;250:81–104. PMID: 2403498.

6. Dinçyürek H, Kocaoglu M, Eralp IL, et al. Functional results of lower extremity lengthening by motorized intramedullary nails. Acta Orthop Traumatol Turc 2012;46(1):42–49. DOI: 10.3944/aott.2012.2671.

7. Dahl MT, Gulli B, Berg T. Complications of limb lengthening. A learning curve. Clin Orthop Relat Res 1994;301:10–18. PMID: 8156659.

8. Black SR, Kwon MS, Cherkashin AM, et al. Lengthening in congenital femoral deficiency: A comparison of circular external fixation and a motorized intramedullary nail. J Bone Joint Surg Am 2015;97(17):1432–1440. DOI: 10.2106/JBJS.N.00932.

9. Sabharwal S. CORR Insights®: What are the potential benefits and risks of using magnetically driven antegrade intramedullary lengthening nails for femoral lengthening to treat leg length discrepancy? Clin Orthop Relat Res 2022;480(4):804–806. DOI: 10.1097/CORR.0000000000002079.

10. Cherkashin AM, Samchukov ML, Birch JG, et al. Evaluation of complications of treatment of severe Blount’s disease by circular external fixation using a novel classification scheme. J Pediatr Orthop Part B 2015;24(2):123–130. DOI: 10.1097/BPB.0000000000000138.

11. Garbuz DS, Masri BA, Esdaile J, et al. Classification systems in orthopaedics. J Am Acad Orthop Surg 2002;10(4):290–297. DOI: 10.5435/00124635-200207000-00007.

12. Klein D. Implementing a general framework for assessing interrater agreement in stata. Stata J 2018;18(4):871–901. DOI: 10.1177/1536867X1801800408.

13. Gwet KL. Testing the difference of correlated agreement coefficients for statistical significance. Educ Psychol Meas 2016;76(4):609–637. DOI: 10.1177/0013164415596420.

14. Svanholm H, Starklint H, Gundersen HJG, et al. Reproducibility of histomorphologic diagnoses with special reference to the kappa statistic. APMIS 1989;97(8):689–698. DOI: 10.1111/j.1699-0463.1989.tb00464.x.

15. Clavien PA, Sanabria JR, Strasberg SM. Proposed classification of complications of surgery with examples of utility in cholecystectomy. Surgery 1992;111(5):518–526. PMID: 1598671.

16. Feinstien AR, Cicchetti D V. High agreement but low kappa. J Clin Epidemiol 1990;43(6):551–558. DOI: 10.1016/0895-4356(90)90159-m.